Assessing a Goodness of Prediction of a Model in a Sensor System

The present disclosure generally relates to a sensor system, and more particularly, to enhancing the evaluation of predictive power in predictive models in a sensor system via a goodness of prediction evaluation.

A variety of approaches and instruments may be used in predictive analytics and model evaluation techniques to examine past data and forecast future trends or events. Large datasets are analyzed using algorithms, machine learning models, and data mining techniques to find patterns and relationships. To make sure that predictive models produce solid and significant findings, model assessment approaches are essential for evaluating the performance, accuracy, and dependability of these models.

According to an embodiment, one or more a non-transitory computer readable storage media store program instructions which, when executed by a processor, causes the processor to perform a procedure including extracting raw data with a sensor array of a system, obtaining a pool of predictive models, for each predictive model of the pool of predictive models, generating a square cross-validated correlation (SCVC) using the raw data as predictors, and for a predictive model with the highest SCVC, generating a proportional loss in predictive power (PLPP). In response to the PLPP of the predictive model with the highest SCVC meeting a predetermined pass criteria, the predictive model with the highest SCVC is deployed as a principal predictive model to provide an early alert about a failing parameter of the system when new raw data is generated by the sensor array.

According to an embodiment, a system includes a sensor array that includes a plurality of sensors. The system also includes a processor configured to extract raw data from the sensor array, obtain a pool of predictive models, and for each predictive model of the pool of predictive models, generate a square cross-validated correlation (SCVC) using the raw data as predictors. For a predictive model with the highest SCVC, the processor generates a proportional loss in predictive power (PLPP), and responsive to the PLPP of the predictive model with the highest SCVC meeting a predetermined pass criteria, the processor deploys the predictive model with the highest SCVC as a principal predictive model to provide an early alert about a failing parameter of the system when new raw data is generated by the sensor array.

According to an embodiment of the present disclosure, a method is disclosed to extract raw data from a sensor array of a system, obtain a pool of predictive models, and for each predictive model of the pool of predictive models, generate a square cross-validated correlation (SCVC) using the raw data as predictors. For a predictive model with the highest SCVC, the method generates a proportional loss in predictive power (PLPP), and responsive to the PLPP of the predictive model with the highest SCVC meeting a predetermined pass criteria, the method deploys the predictive model with the highest SCVC as a principal predictive model to provide an early alert about a failing parameter of the system when new raw data is generated by the sensor array.

In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it should be apparent that the present teachings may be practiced without such details. In other instances, well-known methods, procedures, components, and/or circuitry have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

The illustrative embodiments are related to systems and methods for selecting a principal predictive model for a sensor system. The sensor system may be a physical system comprising a sensor array configured to measure raw data for an alert mechanism that is adapted to providing an early alert about a failing parameter of the sensor system. The sensor system may be a system for product manufacturing, medical testing, physical/biological/chemical quality analyses, product authenticity and combinations thereof.

The illustrative embodiments recognize that in many systems, a common assessment of predictive power for a classical linear regression model is the square population cross-validated multiple correlation (SCVC). The SCVC may be used for discriminating or ranking models selected from a pool of candidate models on the basis of their prediction capabilities. Models associated with larger SCVC values have higher predictive power and may be desired. Formally, the (sample) SCVC of an estimated or fitted regression equation with coefficient vector estimate {circumflex over (β)}, is an unknown constant parameter denoted by

It is a realized value of a random parameter which is the population SCVC. Methods for estimating

2 exist. However, a maximum achievable value is the square population multiple correlation, ρ, which is an unknown parameter. Therefore, it may be difficult to assess the magnitude

2 2 2 relative to ρwithout knowledge of ρ. For example, when a final model is built through a model selection process using the SCVC as the selection criterion the final model's (estimated) SCVC may be devoid of information about its magnitude relative to ρ. It may only suggest that the final model has the largest SCVC among a pool of candidate models. Consequently, it may be challenging to accurately assess the model predictive power on the basis of the magnitude of its estimated SCVC alone. Any careless assessment may yield wrong decisions whereby a moderately small SCVC is misinterpreted as a low predictive power and a moderately large SCVC is mistaken for a high predictive power.

In embodiments herein, applications of the system generate a goodness of prediction assessment to supplement a square cross-validated correlation (SCVC) of an estimated linear regression equation. More particularly, embodiments utilize the SCVC as a model selection criterion to choose the best predictive linear model from a pool of candidate models and employ the goodness of prediction assessment to appraise the predictive power of the final predictive linear model for potential deployment. The goodness of prediction assessment is an estimator of the proportional loss in predictive power (PLPP) the estimated linear regression induces when the estimated linear regression, rather than the true model, is used to make predictions over new samples. Further, and equivalently, a complement statistic, 1−PLPP, assesses the magnitude of the estimated linear regression model SCVC relative to its maximum achievable value, the unknown square population multiple correlation.

In embodiments, a system includes a sensor array includes a plurality of sensors. The system also includes a processor configured to extract raw data from a sensor array of the system, obtain a pool of predictive models, and for each predictive model of the plurality of predictive models, generate a square cross-validated correlation (SCVC) using the raw data as predictors. For a principal predictive model with the highest SCVC, the processor of the system is configured to generate a proportional loss in predictive power (PLPP), and responsive to the PLPP meeting a predetermined pass criteria, deploy the principal predictive model to provide an early alert about a failing parameter of the system when new raw data is generated by the sensor array.

In embodiments, the sensor array comprises dedicated sensors in a manufacturing system or other target physical system configured to accurately measure physical, biological or chemical properties of materials (such as raw materials or the ambient environment, including, for example, mass, weight, chemical composition, equipment information, etc.) which may be used in modeling an alarm system to provide an early alert about a failing parameter related to the system (such as a predicted viscosity or water content of a final product being too low). The sensors are not generic computers and produce measurements as raw data. Measurements as used herein generally refer to raw data that are precise and accurate values that cannot be obtained with the eye or mind (such as in a gauging or estimation process) but can rather be obtained through the use of the special purpose sensors or measurement tools configured for accurate and precise quantification. By obtaining a plurality of predictive models, the measured raw data may be used to assess corresponding SCVCs and PLPPs of the predictive models, which in conjunction aid to select a principal/optimal model for accurate alerts on new raw data.

In one embodiment, certain operations are described as occurring at a certain component or location. Such locality of operations is not intended to be limiting. Any operation described herein as occurring at or performed by a particular component, can be implemented in such a manner that one component-specific function causes an operation to occur, or be performed, at another component, e.g., at a local or remote engine, respectively. In one embodiment, the method described herein, is implemented to execute on a particularly configured computing device or data processing system and provides substantial advancement of the functionality of that computing device or data processing system. Embodiments thus have the capacity to improve the technical field of performance monitoring and alert systems using measure raw data from a sensor array. For example, as opposed to performing a plurality of computations on a generic standalone computer, the illustrative embodiments can utilize collective decision making in a monitoring/alert system to manage thousands of raw data measurements from a plurality of sensors in real-time with a careful observation and methodology, wherein a goodness of prediction quality optimizes performance of the alert system through automatically and dynamically generating and testing the performance of predictive alerting models. In an example, in response to a challenger model performing better than a principal model, the challenger model is used to replace the principal model for alerts.

Importantly, although the operational/functional descriptions described herein may be understandable by the human mind, they are not abstract ideas of the operations/functions divorced from computational implementation of those operations/functions. Rather, the operations/functions represent a specification for an appropriately configured computing device. As discussed in detail below, the operational/functional language is to be read in its proper technological context, i.e., as concrete specifications for physical implementations.

It should be appreciated that aspects of the teachings herein are beyond the capability of a human mind. It should also be appreciated that the various embodiments of the subject disclosure described herein can include information that is impossible to obtain manually by an entity, such as a human user. For example, the type, amount, and/or variety of information included in performing the process discussed herein can be more complex than information that could be reasonably processed manually by a human user.

The illustrative embodiments are described with respect to certain types of machines. The illustrative embodiments are also described with respect to other scenes, subjects, measurements, devices, data processing systems, environments, components, and applications, by way of example only. Any specific manifestations of these and other similar artifacts are not intended to be limiting to the disclosure. Any suitable manifestation of these and other similar artifacts can be selected within the scope of the illustrative embodiments.

Furthermore, the illustrative embodiments may be implemented with respect to any type of data, data source, or access to a data source over a data network. Any type of data storage device may provide the data to an embodiment of the disclosure, either locally at a data processing system or over a data network, within the scope of the disclosure. Where an embodiment is described using a mobile device, any type of data storage device suitable for use with the mobile device may provide the data to such embodiment, either locally at the mobile device or over a data network, within the scope of the illustrative embodiments.

The illustrative embodiments are described using specific surveys, code, hardware, algorithms, designs, architectures, protocols, layouts, schematics, and tools only as examples and are not limiting to the illustrative embodiments. Furthermore, the illustrative embodiments are described in some instances using particular software, tools, and data processing environments only as an example for the clarity of the description. The illustrative embodiments may be used in conjunction with other comparable or similarly purposed structures, systems, applications, or architectures. For example, other comparable devices, structures, systems, applications, or architectures, therefore, may be used in conjunction with such embodiment of the disclosure within the scope of the disclosure. An illustrative embodiment may be implemented in hardware, software, or a combination thereof.

The examples in this disclosure are used only for the clarity of the description and are not limiting to the illustrative embodiments. Additional data, operations, actions, tasks, activities, and manipulations will be conceivable from this disclosure and the same are contemplated within the scope of the illustrative embodiments.

Any advantages listed herein are only examples and are not intended to be limiting to the illustrative embodiments. Additional or different advantages may be realized by specific illustrative embodiments. Furthermore, a particular illustrative embodiment may have some, all, or none of the advantages listed above

1 FIG. 2 FIG. 1 FIG. 2 FIG. With reference to the figures and in particular with reference toand, these figures are example diagrams of data processing environments in which illustrative embodiments may be implemented.andare only examples and are not intended to assert or imply any limitation with regard to the environments in which different embodiments may be implemented. A particular implementation may make many modifications to the depicted environments based on the following description.

1 FIG. 100 100 102 102 100 102 depicts a block diagram of a network of data processing or sensor systems in which illustrative embodiments may be implemented. Data processing environmentis a network of computers in which the illustrative embodiments may be implemented. Data processing environmentincludes network. Networkis the medium used to provide communications links between various devices and computers connected together within data processing environment. Networkmay include connections, such as wire, wireless communication links, or fiber optic cables.

102 104 106 102 108 100 110 112 114 102 104 106 110 112 114 104 Clients or servers are only example roles of certain data processing systems connected to networkand are not intended to exclude other configurations or roles for these data processing systems. Serverand servercouple to networkalong with storage unit. Software applications may execute on any computer in data processing environment. Client, client, clientare also coupled to network. A data processing system, such as serveror server, or clients (client, client, client) may contain data and may have software applications or software tools executing thereon. Servermay include one or more GPUs (graphics processing units) for training one or more models.

1 FIG. 104 106 110 112 114 Only as an example, and without implying any limitation to such architecture,depicts certain components that are usable in an example implementation of an embodiment. For example, servers and clients are only examples and not to imply a limitation to a client-server architecture. As another example, an embodiment can be distributed across several data processing systems and a data network as shown, whereas another embodiment can be implemented on a single data processing system within the scope of the illustrative embodiments. Data processing systems (server, server, client, client, client) also represent example nodes in a cluster, partitions, and other configurations suitable for implementing an embodiment.

120 120 110 120 120 1 FIG. 1 FIG. Deviceis an example of a device described herein. For example, devicecan take the form of a smartphone, a special purpose fabrication platform, a tablet computer, a laptop computer, clientin a stationary or a portable form, a wearable computing device, or any other suitable device. Any software application described as executing in another data processing system incan be configured to execute in devicein a similar manner. Any data or information stored or produced in another data processing system incan be configured to be stored or produced in devicein a similar manner.

128 124 128 128 118 108 Predictive power evaluation enginemay execute as part of predictive power evaluation system, or on any data processing system herein. Predictive power evaluation enginemay also execute as a cloud service communicatively coupled to system services, hardware resources, or software elements described herein. Predictive power evaluation enginemay be operable to extract raw data generated by the sensor array for use in generating a principal predictive model configured to provide an early alert about a failing parameter of a system. Databaseof storage unitstores one or more measurements or data from a sensor or sensor array in repositories for computations herein.

116 116 108 116 116 104 106 110 112 114 122 110 104 Server applicationimplements an embodiment described herein. Server applicationcan use data from storage unitfor computations herein. Server applicationcan also obtain data from any client for computations. Server applicationcan also execute in any of data processing systems (serveror server, client, client, client), such as client applicationin clientand need not execute in the same system as server.

104 106 108 110 112 114 120 102 110 112 114 Server, server, storage unit, client, client, client, devicemay couple to networkusing wired connections, wireless communication protocols, or other suitable data connectivity. Client, clientand clientmay be, for example, personal computers or network computers.

104 110 112 114 110 112 114 104 110 112 114 100 104 116 106 118 124 128 124 128 In the depicted example, servermay provide data, such as boot files, operating system images, and applications to client, client, and client. Client, clientand clientmay be clients to serverin this example. Client, clientand clientor some combination thereof, may include their own data, boot files, operating system images, and applications. Data processing environmentmay include additional servers, clients, and other devices that are not shown. Serverincludes a server applicationthat may be configured to implement one or more of the functions described herein in accordance with one or more embodiments. Servermay include a configuration to aggregate sensor measurements for storage in database. An operator of the predictive power evaluation systemcan include individuals, computer applications, and electronic devices. The operators may employ the predictive power evaluation engineof the predictive power evaluation systemto make predictions or decisions about a failing parameter. An operator may desire that the predictive power evaluation engineperform methods to satisfy a predetermined evaluation criteria.

100 102 100 1 FIG. The data processing environmentmay also be the Internet. Networkmay represent a collection of networks and gateways that use the Transmission Control Protocol/Internet Protocol (TCP/IP) and other protocols to communicate with one another. At the heart of the Internet is a backbone of data communication links between major nodes or host computers, including thousands of commercial, governmental, educational, and other computer systems that route data and messages. Of course, data processing environmentalso may be implemented as a number of different types of networks, such as for example, an intranet, a local area network (LAN), or a wide area network (WAN).is intended as an example, and not as an architectural limitation for the different illustrative embodiments.

100 100 100 Among other uses, data processing environmentmay be used for implementing a client-server environment in which the illustrative embodiments may be implemented. A client-server environment enables software applications and data to be distributed across a network such that an application functions by using the interactivity between a client data processing system and a server data processing system. Data processing environmentmay also employ a service-oriented architecture where interoperable software components distributed across a network may be packaged together as coherent business applications. Data processing environmentmay also take the form of a cloud, and employ a cloud computing model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service.

2 FIG. 1 FIG. 200 104 106 110 112 114 120 124 With reference to, this figure depicts a block diagram of a data processing system in which illustrative embodiments may be implemented. Data processing systemis an example of a computer, such as server, server, client, client, client, device, or predictive power evaluation systemin, or another type of device in which computer usable program code or instructions implementing the processes may be located for the illustrative embodiments.

200 200 120 200 200 200 1 FIG. Data processing systemis also representative of a data processing system or a configuration therein in which computer usable program code or instructions implementing the processes of the illustrative embodiments may be located. Data processing systemis described as a computer only as an example, without being limited thereto. Implementations in the form of other devices, such as devicein, may modify data processing system, such as by adding a touch interface, and even eliminate certain depicted components from data processing systemwithout departing from the general description of the operations and functions of data processing systemdescribed herein.

200 202 204 206 208 210 202 206 206 210 202 In the depicted example, data processing systememploys a hub architecture including North Bridge and memory controller hub (NB/MCH)and South Bridge and input/output (I/O) controller hub (SB/ICH). Processing unit, main memory, and graphics processorare coupled to North Bridge and memory controller hub (NB/MCH). Processing unitmay contain one or more processors and may be implemented using one or more heterogeneous processor systems. Processing unitmay be a multi-core processor. Graphics processormay be coupled to North Bridge and memory controller hub (NB/MCH)through an accelerated graphics port (AGP) in certain implementations.

212 204 216 220 222 224 232 234 204 218 226 230 204 228 234 224 226 230 236 204 218 a a In the depicted example, local area network (LAN) adapteris coupled to South Bridge and input/output (I/O) controller hub (SB/ICH). Audio adapter, keyboard and mouse adapter, modem, read only memory (ROM), universal serial bus (USB) and other ports, and PCI/PCIe devicesare coupled to South Bridge and input/output (I/O) controller hub (SB/ICH)through bus. Hard disk drive (HDD) or solid-state drive (SSD)and CD-ROMare coupled to South Bridge and input/output (I/O) controller hub (SB/ICH)through bus. PCI/PCIe devicesmay include, for example, Ethernet adapters, add-in cards, and PC cards for notebook computers. PCI uses a card bus controller, while PCIe does not. Read only memory (ROM)may be, for example, a flash binary input/output system (BIOS). Hard disk drive (HDD) or solid-state drive (SSD)and CD-ROMmay use, for example, an integrated drive electronics (IDE), serial advanced technology attachment (SATA) interface, or variants such as external-SATA (eSATA) and micro-SATA (mSATA). A super I/O (SIO) devicemay be coupled to South Bridge and input/output (I/O) controller hub (SB/ICH)through bus.

208 224 226 230 a Memories, such as main memory, read only memory (ROM), or flash memory (not shown), are some examples of computer usable storage devices. Hard disk drive (HDD) or solid-state drive (SSD), CD-ROM, and other similarly usable devices are some examples of computer usable storage devices including a computer usable storage medium.

206 200 200 2 FIG. An operating system runs on processing unit. The operating system coordinates and provides control of various components within data processing systemin. The operating system may be a commercially available operating system for any type of computing platform, including but not limited to server systems, personal computers, and mobile devices. An object oriented or other type of programming system may operate in conjunction with the operating system and provide calls to the operating system from programs or applications executing on data processing system.

116 122 226 226 208 206 206 208 224 1 FIG. b a Instructions for the operating system, the object-oriented programming system, and applications or programs, such as server applicationand client applicationin, are located on storage devices, such as in the form of codeson Hard disk drive (HDD) or solid-state drive (SSD), and may be loaded into at least one of one or more memories, such as main memory, for execution by processing unit. The processes of the illustrative embodiments may be performed by processing unitusing computer implemented instructions, which may be located in a memory, such as, for example, main memory, read only memory (ROM), or in one or more peripheral devices.

226 214 214 214 214 226 214 214 214 214 b a b c d b a b c d. Furthermore, in one case, codemay be downloaded over networkfrom remote system, where similar codeis stored on a storage devicein another case, codemay be downloaded over networkto remote system, where downloaded codeis stored on a storage device

1 FIG. 2 FIG. 1 FIG. 2 FIG. The hardware inandmay vary depending on the implementation. Other internal hardware or peripheral devices, such as flash memory, equivalent non-volatile memory, or optical disk drives and the like, may be used in addition to or in place of the hardware depicted inand. In addition, the processes of the illustrative embodiments may be applied to a multiprocessor data processing system.

200 In some illustrative examples, data processing systemmay be a personal digital assistant (PDA), which is generally configured with flash memory to provide non-volatile memory for storing operating system files and/or user-generated data. A bus system may comprise one or more buses, such as a system bus, an I/O bus, and a PCI bus. Of course, the bus system may be implemented using any type of communications fabric or architecture that provides for a transfer of data between different components or devices attached to the fabric or architecture.

208 202 A communications unit may include one or more devices used to transmit and receive data, such as a modem or a network adapter. A memory may be, for example, main memoryor a cache, such as the cache found in North Bridge and memory controller hub (NB/MCH). A processing unit may include one or more processors or CPUs.

1 FIG. 2 FIG. 200 The depicted examples inandand above-described examples are not meant to imply architectural limitations. For example, data processing systemalso may be a tablet computer, laptop computer, or telephone device in addition to taking the form of a mobile or wearable device.

200 200 206 206 208 208 226 226 200 a a Where a computer or data processing system is described as a virtual machine, a virtual device, or a virtual component, the virtual machine, virtual device, or the virtual component operates in the manner of data processing systemusing virtualized manifestation of some or all components depicted in data processing system. For example, in a virtual machine, virtual device, or virtual component, processing unitis manifested as a virtualized instance of all or some number of hardware processing unitsavailable in a host data processing system, main memoryis manifested as a virtualized instance of all or some portion of main memorythat may be available in the host data processing system, and Hard disk drive (HDD) or solid-state drive (SSD)is manifested as a virtualized instance of all or some portion of Hard disk drive (HDD) or solid-state drive (SSD)that may be available in the host data processing system. The host data processing system in such cases is represented by data processing system.

3 FIG. 124 308 308 Turning now to, a block diagram of a predictive power evaluation systemfor generating a principal predictive modelis disclosed. The principal predictive modelmay be generated for a process such as a product manufacturing process, a medical testing process, a physical/biological/chemical quality analysis process, a product authenticity process, other similar processes involving the use of a sensors to measure and obtain raw data, and combinations thereof.

3 FIG. is described in conjunction with an example product manufacturing process that involve batch manufacturing and is not meant to be limiting as other examples may be obtained in view of the descriptions herein. Batch manufacturing may be considered as the process of manufacturing that takes raw materials and uses a formula or recipe to combine and refine the raw materials by flowing through a process that results in a final product. Examples of materials made through batch manufacturing include food and beverage products, pharmaceuticals, and chemicals. Batch manufacturing can be contrasted with discrete manufacturing where distinct, countable parts are made and often assembled together into a final product.

302 126 304 In the manufacturing process, final products may be manufactured in batches. Samples are taken from each batch and properties of the final products measured to obtain raw data about whether the final product conforms to specification. For example, in the manufacture of asphalt, raw materials include aggregates such as gravel, slag, rock, or recycled material. The materials go through several process steps, including cooling, heating, and mixing, according to a recipe. Environmental measurements during the processing of the raw materials such as temperature and humidity, process measurements related to the raw materials such as line speed and pressure, direct physical, chemical or biological measures of raw material measurements such as weight, and chemical composition are taken by sensorsof the sensor arrayas raw datawhile each batch moves through the process steps.

306 308 310 Following the manufacturing process, samples from each batch may be used for further measurements to determine actual final specifications such as a final viscosity or water content of the final product. With such traceability, predictive modelscan be generated and a goodness of prediction of the predictive models assessed to provide a principal predictive modeland/or challenger predictive modelsthat can be used to generate an early alert that a metric of a final product is likely to be out of specification based on current process conditions.

306 128 306 More specifically, the predictive models, which can be regression models, can take measurements from a quality assessment of the final products and use previously collected measurements related to the raw materials that were used to produce the final products as the predictors. Along with generating a squared cross-validated correlation (SCVC), a proportional loss in predictive power (PLPP), described in more detail herein, is also generated by the predictive power evaluation engineto provide a complete and accurate assessment of the prediction capability of the resulting predictive models. In the asphalt manufacturing example, input variables related to raw materials, such as quantity, flow, pH, temperature and pressure all potentially have an impact on the viscosity of the final product. By tracking these process inputs, it is possible to predict in advance that the viscosity of the final product is likely to fall out of specification and make adjustments before such defects occur.

306 306 306 306 306 306 306 306 306 306 2 2 For a given predictive model, the PLPP measures the loss in predictive power the predictive modelinduces when the predictive model, rather than a true regression model is used to make predictions over new observations. Equivalently the complement RCSCV (which is 1−PLPP), assesses a magnitude of the SCVC relative to its unknown maximum achievable value, the square population multiple correlation (ρ). Thus, the PLPP is a useful metric for assessing the deployment worthiness after the SCVC of a predictive modelhas been estimated. A low PLPP suggests that the estimated predictive modelinduces a low loss in predictive power when the predictive model, rather than the true model is used to make predictions on new data. In other words, the SCVC is large relative to its maximum possible value so that the estimated predictive modelcaptures the underlying patterns in the data well and can make accurate predictions on new data. When comparing multiple predictive model, the PLPP of the predictive modelwith the highest SCVC is estimated to see if that predictive modelcan be generalized to new samples. Table 1 provides rough guidelines for classifying the magnitude of the SCVC relative to its maximum possible value ρ, based on the magnitude of the PLPP of a linear regression model.

2 Table 1. Rough guidelines for classifying the magnitude of the SCVC relative to its maximum possible value, ρ, based on the magnitude of the PLPP or RSCVC of a linear regression model.

Magnitude of SCVC PLPP RSCVC 2 relative to ρ c 0.5 ≤ δ({circumflex over (β)}) < 1.0 c 0 < 1 − δ({circumflex over (β)}) ≤ 0.5 Small c 0.3 ≤ δ({circumflex over (β)}) < 0.5 c 0.5 < 1 − δ({circumflex over (β)}) ≤ 0.7 Moderately small c 0.2 ≤ δ({circumflex over (β)}) < 0.3 c 0.7 < 1 − δ({circumflex over (β)}) ≤ 0.8 Moderately large c 0.1 ≤ δ({circumflex over (β)}) < 0.2 c 0.8 < 1 − δ({circumflex over (β)}) ≤ 0.9 Large c 0 ≤ δ({circumflex over (β)}) < 0.1 c 0.9 < 1 − δ({circumflex over (β)}) ≤ 1 Very Large

308 310 310 308 310 310 124 308 306 In an illustrative embodiment, principal predictive modelis the current best-performing model, selected using the SCVC as the discriminating criteria in a forward selection process. Models with higher SCVC values are preferred. Challenger predictive modelsmay be competing models. Each challenger predictive modelmay have the same response variable as the principal predictive model. However, challenger predictive modelscan have different predictors. The challenger predictive modelthat generates the best results in the predictive power evaluation systemcan become the new principal predictive modelif promoted. In essence, each SCVC estimate can be followed by a corresponding PLPP estimate to determine if the predictive modelis worth deploying.

304 308 308 124 For example, using a model for raw dataabout asphalt, a principal predictive modelwas generated to be two-predictor model with an estimated SCVC value of 0.2635, a first challenger predictive model (Challenger 1) was a 3-predictor model with an estimated SCVC value of 0.2441, and a second challenger predictive model (Challenger 2) was a 4-predictor model with an estimated SCVC value of 0.2400. In addition, the estimated PLPP of the principal predictive model was 0.0217 indicating that there is only about a 2.2% loss in predictive power when the principal predictive model, rather than the corresponding true model, is used to make predictions. Therefore, the principal predictive modelof the predictive power evaluation systemsis deployable. Similarly, the PLPP of the challenger predictive models were 0.0457 and 0.0669 respectively so that there are only about 4.6% and 6.7% losses in predictive powers when the challenger predictive models are used in new samples to make predictions. Thus, the challenger predictive models may in some cases be deployable as well.

308 124 100 308 310 304 308 In illustrative embodiments, once a principal predictive modelis deployed into production, it may be monitored to detect when conditions have changed such that an update to the existing model can be performed. To that end, the principal predictive model and challenger predictive models can be refit with the latest data on a scheduled basis in the predictive power evaluation systemor data processing environment(e.g. a manufacturing plant or a regulatory system), such as once per week or once per month. Measures of predictive power including the SCVC and PLPP are then computed. First, the SCVC for the principal predictive modeland challenger predictive modelsare estimated using the latest raw data. The estimated SCVCs are then compared to determine whether any challenger model has better SCVC estimate than the principal predictive model. In addition, the PLPP of the models are examined to determine if they maintain their respective deployable status. Over time, models with consistently low SCVC and high PLPP might need adjustment or redevelopment.

4 FIG. 402 418 124 304 126 402 306 406 306 402 414 308 402 408 402 308 418 126 illustrates an applicationfor generating alertsin a predictive power evaluation system. The application receives or extracts or generates raw datafrom or using on the sensor array. The applicationobtains a pool of predictive modelsfor modeling by the modeler. For each predictive modelof the pool of predictive models, the applicationgenerates, by the SCVC generator, a square cross-validated correlation (SCVC) using the raw data as predictors. For a principal predictive modelwith the highest SCVC, the applicationgenerates, by the PLPP generator, a proportional loss in predictive power (PLPP). Responsive to the PLPP meeting a predetermined pass criteria (such as being in a predetermined range) the applicationdeploys the principal predictive modelto provide an early alertabout a failing parameter of the system when new raw data is generated by the sensor array.

408 410 408 412 5 FIG. 6 FIG. The PLPP generatorcan generate the PLPP as a point estimateas illustrated in. The PLPP generatorcan also or alternatively generate the PLPP as a confidence interval, as shown in.

5 FIG. is a flowchart illustrating a routine for generating the PLPP as a point estimate. Firstly, as discussed earlier, the generation of the PLPP may be performed after generating the SCVC. The SCVC is a common assessment of predictive power for a classical linear regression model. The SCVC can be used for ranking models on the basis of their prediction capabilities. Models associated with larger SCVC values have higher predictive power and may be desired. Formally, the (sample) SCVC of an estimated or fitted regression equation with coefficient vector estimate {circumflex over (β)}, is an unknown constant parameter denoted by

It is a realized value of a random parameter which is the population SCVC. Methods for estimating

2 exist. However, a maximum achievable value is the square population multiple correlation, ρ, which is an unknown parameter. Therefore, it may be difficult to assess the magnitude

2 2 2 relative to ρwithout knowledge of ρ. For example, when a final model is built through a model selection process using the SCVC as the selection criterion the final model's (estimated) SCVC may be devoid of information about its magnitude relative to ρ. It may only suggest that the final model has the largest SCVC among a pool of candidate models. Consequently, the predictive power of models can be assessed not only on the basis of the magnitude of its estimated SCVC but in conjunction with the PLPP.

A common practice to assessing the closeness of

2 2 to its maximum achievable value ρis to calculate the amount of shrinkage of the sample coefficient of determination, R, relative to R. Typically, when the estimated regression equation is used to make predictions over new samples the sample coefficient of determination shrinks to the estimated sample SCVC,

2 2 2 The amount of shrinkage of Rrelative to Ror the proportional shrinkage of Ris therefore obtained as

2 Similarly, the estimated regression equation induces a loss in predictive power when it, rather than the true regression equation, is used to make predictions over new samples. In other words, the population coefficient of determination, ρ, shrinks to become

2 when the estimated regression equation is used to make prediction over new samples. The amount of loss in predictive power relative to ρor the proportional loss in predictive power (PLPP) associated with the estimated regression equation is given by

Thus, the method of assessing the closeness of

2 to ρ(or assessing the magnitude of

2 2 relative to ρ) using the proportional shrinkage of Ris equivalent to estimating the sample

Irrespective of the method for estimation of

2 2 2 2 2 2 2 2 2 2 2 however, a limitation of R: the proportional shrinkage of approach is a gross overestimation of the sample PLPP, to some extent, due to Ralways overestimating ρ. Specifically, simulation results reveal that the proportional shrinkage of Rseverely overestimates the PLPP for regression samples drawn from population with small ρvalues (ρ<0.5) or for small and moderate regression samples where the ratio of the sample size to the number of predictors is small or moderate. As a result of this overestimation, a fitted regression equation with a low PLPP (or equivalently large SCVC relative to ρ) will be misinterpreted as having a large PLPP (or equivalently small SCVC relative to ρ) and therefore would not be deployed by mistake. Such misleading results may have tremendous consequences. Since in most meaningful practical applications the magnitude of ρis not known these results suggest that R, the proportional shrinkage method for estimating the PLPP or equivalently for estimating the magnitude of the SCVC relative to its maximum possible value, ρ, may be discontinued.

However, based on the same definitions and notations, the sample PLPP of an estimated regression equation with coefficient vector estimate {circumflex over (β)} is given by

This assesses the closeness of the sample SCVC,

2 to its maximum achievable value, ρ. Thus, a small value of the PLPP close to 0 indicates that there may only be a little loss in predictive power when the estimated regression equation, instead of the true regression equation, is used to make predictions over new samples. A large value of the PLPP close to 1, on the other hand, can indicate that the estimated regression model causes a great loss in prediction power when it is used, in place of the true equation, to make predictions over new samples.

2 An alternative but directly equivalent measure to the PLPP is the RSCVC, the ratio of the SCVC to ρgiven by

The RSCV assesses the magnitude of

2 2 2 2 2 relative to ρ, and can be expressed in terms of the PLPP as RSCVC=1−PLPP. As a result, the PLPP and RSCVC have the same statistical properties. In embodiments, interpreting a large value of a statistic as a positive outcome may utilize the RSCVC to measure the closeness of the SCVC to ρ. Specifically, if an estimated regression equation RSCVC is close to 1 (SCVC is close to ρ) then the SCVC is large relative to ρ; on the other hand, if the RSCVC is close to 0 then SCVC is small compared to ρ.

c The population PLPP is a random parameter whose distribution can be expressed in terms noncentral beta distributions. A new point estimator of sample PLPP, δ({circumflex over (β)}), derived from the mean of the distribution of the population PLPP is provided. In addition, an algorithm for finding a confidence interval for the average PLPP for a given fitted regression equation is also provided.

c c c c For a given estimated regression equation with coefficient estimates {circumflex over (β)}, an analytical point estimator for δ({circumflex over (β)}) is provided and standard error of estimates derived herein. A confidence interval method for the average of Δ(the mean population PLPP) is also provided. These methods are based upon the mean of the distribution of Δ. The exact mean of Δ, however, has an opened expression that involves infinite power series. Fortunately, the exact statistical mean is optional for estimation purposes. Typically, a first or second order approximation of the exact mean is suitable. For example, Browne (1975) estimator is derived from a first order approximation of the mean of the population SCVC. Similarly, a point estimator of a regression sample PLPP can be derived from a first order approximation of the exact mean. Confidence intervals, however, can be derived from a second order approximation of the exact mean to improve on accuracy of the intervals.

By the δ-method, a first and second order approximations for the mean of the ratio of two random variables, U and V, are given as follows:

c This result is obtained by taking the expectation of the second order Taylor expansion of the function g(U, V)=U/V about the point [E(U), E(V)]. In addition, the distribution of Δcan be expressed in terms of independents noncentral chi square and central chi square distribution as follows:

where

independent of

2 1 2 Letting U=and V=Y+Y, yields the following:

Thus, first order approximation of the mean PLPP is as follows:

The second order approximation is given as follows:

2 2 2 2 1 2 In the above, λ=(n−p−2)ρ/(1−ρ). It can be easily shown that both {tilde over (μ)}and {tilde over (μ)}are monotone decreasing functions of ρ. Like the exact mean, they decrease from (p−1)/p to 0 as ρvaries from 0 to 1. In addition, using the δ-method, an approximate variance of the aforementioned function g(U, V)=U/V is given as follows:

2 1 2 1 2 2 c As before, by letting U=Yand V=Y+Y, and using the fact that Yand Yare independent it can be obtained that E(U)=p−1 E(V)=λ+p, Var(V)==2(2λ+p), Cov(U, V)=Var(U)=Var(Y)=2(p−1). After some algebra, an approximate variance for the distribution of Δis given by the following equation:

c Taking the square root yields an approximate standard deviation of the distribution of Δgiven as follows:

For a given estimated regression equation or {circumflex over (β)}, Brown (1975) proposed an estimator of

c c c c c derived from an estimate of the mean of the population square cross-validated correlation,. A similar approach can be used to derive a point estimator of δ({circumflex over (β)}) based on the mean of Δ. For point estimation purposes a point estimator of δ({circumflex over (β)}) can be derived from a first order approximation of the mean of Δbecause it involves a simple inverse function of the non-centrality parameter, λ. More specifically, a point estimator of δ({circumflex over (β)}) may be given by the following equation:

where

2 is an approximate unbiased estimator for ρ(Cattin, 1980). It is given as follows.

2 where Ris the usual coefficient of determination or the sample square multiple correlation. The point estimate

2 2 can be negative in some small samples designs where the population square correlation, ρ, is close to zero. When this occurs, the sample PLPP is estimated as its highest possible average value, (p−1)/p. An alternative estimator can be based on the adjusted R, we denote by

as opposed to

An expression of

is:

From simulation studies, however, the estimator based on

yields better results in small sample designs. In larger sample designs the two estimators yield essentially the same results.

In addition, a large sample approximate standard deviation of the estimator is given as follows:

where

c c c 2 In the above, k=n−p−2. The point estimate, {circumflex over (δ)}, is guaranteed to be in the unit interval [0, 1) just like the true parameter, δ({circumflex over (β)}). In addition, a common conventional estimator of δ({circumflex over (β)}) is the proportional shrinkage of Rgiven by

where

is an estimator of the sample SCVC,

Data splitting cross-validation and analytical (formula-based) estimation methods for the sample SCVC,

c 2 are available. Since the analytical estimation methods have been shown to be superior to data splitting methods, analytical methods can be considered. Among these analytical methods, Browne (1975) estimator is ideal. Thus, the first traditional estimator for δ({circumflex over (β)}) that may be considered is the proportional shrinkage of Rbased on Browne SCVC estimator. The estimator is denoted by BRPS and is given as follows:

The proportional shrinkage estimation method based on Lord (1950) and Nicholson (1960) estimator for

c 2 can be considered. This estimation method for δ({circumflex over (β)}) is considered primarily because sample size planning method, a so-called PEAR method, uses this particular proportional shrinkage form. Specifically, the proportional shrinkage of Rbased on Lord and Nicholson estimator for

is given as follows:

where Lord-Nicholson estimator of the sample SCVC is given

5 FIG. 500 410 500 502 304 126 504 506 2 Accordingly, turning now to, a routinefor generating the point estimateis shown. The routinebegins at blockwherein the input data is generated based on the raw datafrom the sensor array. n is a sample size, and p is the number of predictors. In block, a regression model is fit, and a usual square coefficient of multiple determinations: Ris generated in block. More specifically,

508 In block, Cattin's approximate unbiased estimator of the population square coefficient of multiple determination:

is generated. More specifically,

510 c In block, a point estimate of the non-centrality parameter, λ, for the distribution of Δ, the PLPP of the square population correlation is generated:

512 c c In block, the new point estimate, {circumflex over (δ)}({circumflex over (β)}), of the (sample) PLPP, δ({circumflex over (β)}) is generated. More specifically,

6 FIG. 600 412 is a flowchart illustrating a routinefor generating the PLPP as a confidence intervalin accordance with an illustrative embodiment.

2 2 2 2 Firstly, for a given estimated regression equation or {circumflex over (β)}, an approximate confidence interval (CI) for the unknown mean population PLPP value may be proposed. The CI can be derived from the monotonicity of the mean, μ(ρ, n, p) as a function of ρ. More specifically, for fixed n and p, the mean of μ(ρ, n, p) is a monotone decreasing function of ρ. Therefore, if the interval

2 is a two-sided 100(1−α) percent CI for ρthen

2 is also a two-sided 100(1−α) percent CI for μ(ρ,n,p). For most practical purposes,

may also be replaced with

2 2 2 where {tilde over (μ)}(ρ,n,p) is the second order approximation of μ(ρ,n,p) as derived earlier. In this case, the second order approximation (as opposed to the first order approximation) may be used because it yields better CIs in small sample designs.

2 2 Moreover, approximate CIs for ρ(for sufficiently large samples) are available. These CIs, such as the so-called Helland or Banger and Pammer normal approximation methods can be very sensitive to the normal assumption under which they are derived. Thus, CI methods such as the so-called adjusted F approximation method and adjusted normal approximation method can be used wherein, for a given regression sample with coefficient vector estimate {circumflex over (β)}, an approximate two-sided 100(1−α) percent CI for the mean PLPP can be obtained by firstly generating a two-sided 100(1−α) percent CI for ρusing one of the adjusted approximation methods to obtain

c are generated as the lower and upper limit of the CI, respectively. For most practical purposes, a (one-sided) 100(1−α) percent upper confidence bound for the mean PLPP may be more useful since smaller values of Δare desirable. Therefore, it can be concluded with a 100(1−α) percent confidence that the generated upper bound is the maximum average proportional loss in predictive power for using the estimated regression equation, rather than the true equation, to make predictions over new sample data.

6 FIG. 600 600 602 304 126 604 606 2 Accordingly, turning now to, a routinethat illustrates the generation of the 100(1−α) percent confidence interval is illustrated. The routinebegins at blockwherein the input data is generated using the raw datafrom the sensor array. In blocka regression model is fit, and the usual square coefficient of multiple determinations: Ris generated in block. More specifically,

608 In block, an approximate (1−α)100 percent two-sided CI,

2 for the population square coefficient of multiple determination, ρ, is generated using the adjusted F approximation or the adjusted normal approximation methods, respectively.

2 An approximate (1−α)100 percent two-sided CI for ρmay be given as

where

are the solutions of the following nonlinear equations:

respectively. In the above, the notation

th 1 2 stands for α×100upper percentile point of the F distribution with dand ddegrees of freedom,

1 n y y Also, the quantity {circumflex over (γ)} is the sample (median) kurtosis of the fitted values from the regression model. More specifically, if ŷ, . . . , ŷare the fitted values, {circumflex over ()} the sample mean of the fitted values, and {circumflex over ()} is the sample median of the fitted values of the regression model then {circumflex over (γ)} is calculated as follows:

If the predictor data is known to have come from the multivariate normal distribution, then the true kurtosis γ is 3, so that d={circumflex over (d)}=1. As a result, these equations reduce to the conventional Helland's F approximation CIs.

2 An approximate 100(1−α) percent two-sided CI for ρbased on the adjusted normal approximation method is given by

where

are directly obtained as follows:

α th zis the α×100upper percentile point of the standard normal distribution. If the predictor data is known to have come from the multivariate normal distribution, then the true kurtosis γ is 3, so that d={circumflex over (d)}=1. As a result, these CIs reduce to the conventional normal approximation CIs.

6 FIG. c L U 610 Turning back to, the corresponding CI for the non-centrality parameter, λ, of the distribution of Δis generated as [λ,λ] in block.

More specifically,

612 In block, an approximate (1−α)100 percent two-sided CI for the mean PLPP is generated as

More specifically,

128 408 Using these routings, the predictive power evaluation engine, which may include the PLPP generator, may be used to generate, such as compute, the maximum possible average PLPP with some confidence level. Equivalently, the minimum possible average RSCVC with some confidence level can also be deduced from the PLPP upper confidence bound.

2 306 306 306 The PLPP (or equivalently, the RSCVC), is not used alone as a criterion for comparing candidate models in part because adding predictors to a model tends to increase the PLPP (or equivalently, decrease the RSCVC) of the resulting model. Viewed in this way, the PLPP (or equivalently, the RSCVC) is to quality of prediction what the Rstatistic is to quality of fitness for an estimated regression equation. The PLPP (or the RSCVC), however, can be combined with the SCVC to build models and evaluate their predictive worthiness for deployment purposes. In many applications where the main task consists of choosing the “best” predictive regression model from a pool of candidate predictive modelsthe SCVC may be used as the criterion for comparing the models through a model selection process. The final selected predictive model, though the best predictive modelfrom the pool of candidate models, may not have adequate predictive power for deployment purposes. The SCVC of the final predictive model may not provide enough information for these purposes since its maximum possible value is unknown. The estimated PLPP (or RSCVC) of the final predictive model, however, can be used to accurately assess the predictive worthiness for deployment purposes. Consequently, the PLPP supplements the SCVC.

For small to large sample designs, the assessment of a fitted regression equation PLPP may be based on the new point estimates of the PLPP (or RSCVC) and standard error, or a one-sided upper confidence bound for the mean PLPP (or a one-sided lower confidence bound for the mean RSCVC). For very large sample or big data, the assessment can be based upon point-estimates.

7 FIG. 700 100 700 128 702 304 126 704 128 306 706 128 306 304 708 128 306 710 306 308 304 126 Turning now to, a generalized routinefor providing alerts in the data processing environmentis illustrated. The generalized routinemay be performed by or in conjunction with the predictive power evaluation engine. In block, raw datais extracted from the sensor arrayof an alert or monitoring system. In block, the predictive power evaluation engineobtains a pool of predictive models. In block, the predictive power evaluation enginegenerates an SCVC for each predictive modelof the plurality of predictive models, using the raw dataas predictors. In block, the predictive power evaluation enginegenerates a PLPP for the predictive modelwith the highest SCVC. In block, responsive to the PLPP meeting a predetermined pass criteria, the predictive modelwith the highest SCVC is deployed as a principal predictive modelto provide an early alert about a failing parameter of the system when new raw datais generated by the sensor array.

308 310 310 308 According to illustrative embodiments, another predictive model having a lower SCVC than the SCVC of the principal predictive modelis selected and marked as a challenger predictive model. When the SCVC of the challenger predictive modelbecomes higher than the SCVC of the principal predictive model, and the PLPP of the challenger predictive model meets a predetermined pass criteria, the original principal predictive model can be replaced with the challenger predictive model which becomes the new principal predictive model.

304 304 302 126 According to illustrative embodiments, the raw datais raw datafor a target application selected from the group consisting of product manufacturing or physical products, medical testing, physical/biological/chemical quality analyses, product authenticity and combinations thereof. Further, when the target application is product manufacturing, the raw data can be raw data about a plurality of raw materials, the raw data being representative of measured physical, chemical or biological properties (such as properties of the raw materials or other materials and equipment) in a manufacturing process, and the predictive models provide an early alert about a failing quality parameter of a final product of the product manufacturing. Even further, raw data from one or more sensorsof the sensor arrayis not used for at least one of the predictive models. Therefore, variations in the number of predictors may be obtained for pool of predictive models being assessed.

In further embodiments, the PLPP as a point estimate of a sample PLPP using

c wherein {circumflex over (δ)}({circumflex over (β)}) is the point estimate of the sample PLPP, {circumflex over (λ)} is a point estimate of a non-centrality parameter, and p is a number of predictors. In embodiments, the PLPP can be generated as a (1−α)100 percent lower confidence bound for the mean PLPP using

wherein

U is the (1−α)100 percent lower confidence bound, λis an upper confidence bound for a non-centrality parameter, and p is the number of predictors. In embodiments, the PLPP can be generated as a (1−α)100 percent upper confidence bound for the mean PLPP using

wherein

L is the (1−α)100 percent upper confidence bound, λis a lower confidence bound for a non-centrality parameter, and p is the number of predictors.

In further embodiments, sensors of the sensor array are each configured to measure a physical, biological or chemical property of a material. Each sensor of the sensor array may also measure a different type of raw data.

Any specific manifestations of these and other similar example processes are not intended to be limiting to the invention. Any suitable manifestation of these and other similar example processes can be selected within the scope of the illustrative embodiments.

Thus, a computer implemented method, system or apparatus, and computer program product are provided in the illustrative embodiments for goodness of prediction assessment and other related features, functions, or operations. Where an embodiment or a portion thereof is described with respect to a type of device, the computer implemented method, system or apparatus, the computer program product, or a portion thereof, are adapted or configured for use with a suitable and comparable manifestation of that type of device.

Where an embodiment is described as implemented in an application, the delivery of the application in a Software as a Service (SaaS) model is contemplated within the scope of the illustrative embodiments. In a SaaS model, the capability of the application implementing an embodiment is provided to a user by executing the application in a cloud infrastructure. The user can access the application using a variety of client devices through a thin client interface such as a web browser, or other light-weight client-applications. The user does not manage or control the underlying cloud infrastructure including the network, servers, operating systems, or the storage of the cloud infrastructure. In some cases, the user may not even manage or control the capabilities of the SaaS application. In some other cases, the SaaS implementation of the application may permit a possible exception of limited user-specific application configuration settings.

The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.

Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on a dedicated system or user's computer, partly on the user's computer or dedicated system, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server, etc. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

All features disclosed in the specification, including the claims, abstract, and drawings, and all the steps in any method or process disclosed, may be combined in any combination, except combinations where at least some of such features and/or steps are mutually exclusive. Each feature disclosed in the specification, including the claims, abstract, and drawings, can be replaced by alternative features serving the same, equivalent, or similar purpose, unless expressly stated otherwise.