Systems and methods of implementing scorecards and boosted decision trees

This application claims the benefit of U.S. Provisional Application No. 63/117,940, filed on Nov. 24, 2020, the entirety of which is hereby incorporated herein by reference.

In the current age of technology, machine learning models impact many aspects of everyday life. In many instances, these models are black boxes that are overly complicated and hard to interpret or understand for external reviewers. But as these models become more prevalent, public concern grows about whether such models are making good (for example, sound and fair) decisions. For example, there are various instances of such black box machine learning models generating improper results in prison parole and air quality safety scenarios, among others. Furthermore, there are various regulated industries in which uses of machine learning models may be difficult from a compliance perspective because the company or other entity involved in training and/or operating the model may not be able to effectively explain why the model leads to a particular result in a given circumstance.

Accordingly, improved systems, devices, and methods for efficiently and effectively developing or learning models that are more easily capable of being understood rather than viewed as effectively a black box (and, therefore, easier to review, verify and/or explain with respect to corresponding outputs) are described in further detail herein.

The present disclosure relates to systems and methods of developing scorecards and boosted decision trees that facilitate more robust, and interpretable predictions or decisions relative to existing approaches and systems. More specifically, certain systems and methods described herein employ a scorecard and boosted decision tree tool that can be applied to risk prediction and decision-making problems in finance, medicine, criminal justice, and various other fields.

Systems and methods are described herein for machine learning-based generation of scorecards and boosted decision trees that facilitate explainable predictions. A scorecard machine learning model may be applied to historical records such that the model, for each of a number of variables, automatically generates (a) normal bins for normal values of the variable that fall within a valid range of values and (b) at least one special bin for special values of the variable that fall outside the valid range of values. Adjacent bins of the normal bins may be separated by a threshold value and each normal bin and each special bin may have an assigned score value. A risk assessment score may be generated based at least in part on the model identifying the assigned value score for each of the variables based on the normal or special bin to which each variable is assigned.

A constrained optimization based score card as described herein supports a wide variety of constraints, including a monotonisity constraint, cross variable constraint, and best bin protection zero score constraint. Also described herein are novel penalties or regularizations to significantly improve model robustness and reliability. An EGBDT algorithm as described herein is capable of handling multiple default values, replacing heuristics by start-of-the-art optimization when enforcing monotonicity constraints, and relaxed unnecessary over-constraints in traditional GDBT implementations. The EGBDT algorithm and constrained optimization based score card techniques described herein can be combined together to achieve modeling efficiency and robust performance in production. EGBDT techniques can be applied to performance initial attribute selection and optimal binning. A constrained optimization based score card as described herein can train robust final models based on the outcome from the EGBDT process.

Although certain embodiments and examples are disclosed below, inventive subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and/or uses and to modifications and equivalents thereof. Thus, the scope of the application is not limited by any of the particular embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any particular disclosed sequence. Various operations may be described as multiple discrete operations in turn, in a manner that may be helpful in understanding certain embodiments; however, the order of description should not be construed to imply that these operations are order dependent. Additionally, the structures, systems, and/or devices described herein may be embodied as integrated components or as separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are achieved by any particular embodiment. Thus, for example, various embodiments may be carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other aspects or advantages as may also be taught or suggested herein.

Machine learning and similar models are used in various industries to perform predictive analysis of or on large amounts of data. The machine learning models may involve application of algorithms to inputs, whereby the algorithms generate outputs that generally improve with respect to quality of the outputs and predictive capabilities associated therewith as the algorithms and models are exposed to more input data. In some embodiments, the models may be trained using training or sample data before being applied to generate one or more outputs. Such ML models are often considered to be black boxes that are difficult to understand, in the sense that it may be difficult for an operator of the model (or reviewer of the results of the model) to explain why the model made a certain prediction, classification or produced some other output when given a certain set of input features. For example, it may be difficult to explain to a person what factors caused an ML model to suggest denying the person a loan or what caused the model to output a particular risk score based on a large set of input data regarding the person.

Scorecard models, as described herein, are a class of machine learning models that systems and methods often apply to facilitate quick and interpretable predictions or decisions. While various ML models can provide for risk prediction and solutions to various problems in finance, medicine, criminal justice, and other professional domains, as described above, many of these models are overly complicated and difficult to interpret. The lack of explainability and transparency regarding operation of the ML models (such as multi-layered neural network models that include a large number of weights within multiple layers) and the corresponding systems and methods raises concerns regarding decisions and results reached by the models. The scorecard models described herein provide simple and interpretable models to solve problems described herein while enabling desired problem solving.

Many entities that employ machine learning have been trying to address the concerns of lack of transparency and the like by implementing more explainable artificial intelligence (AI) systems and methods, and more specifically, more explainable ML models. The simple and interpretable ML models enable modelers (and other third party entities) to easily inspect the learnings (for example, patterns learned and applied by) the ML models so that the modelers and others can decide and/or determine whether the outputs generated by the ML models are correct as compared to previous complicated ML models that operated as black boxes and were difficult to inspect.

For example, in a financial setting, a ML model can be applied to a dataset with individuals who recently had defaults on their credit cards to determine a probability that the individual would default on a newly approved credit card. The simple and interpretable ML model applied in this financial data set may generate a result indicating that the individual has a lower probability to default on the newly approved credit card as compared to individuals with no recent delinquencies. The simple and interpretable ML model may arrive at this result based on analyzing similar situations and learning that often credit lenders will only approve a tiny percentage of individuals having recent defaults (such as this consumer) and upon whom the credit lenders have done very thorough inspections and are confident that an additional default is unlikely. The simple and interpretable ML model will allow the modeler or other entity to easily review the analysis and learning of the ML model and identify that the result of the simple and interpretable ML model is accurate based on input data. Thus, if the simple and interpretable ML model was generating incorrect outputs, the modeler could identify how to impose standard regularization penalties or constraints to improve or limit the outputs of the simple and interpretable ML model. On the other hand, the complicated ML model may arrive at this same erroneous result based on picking up a false pattern from the modeling dataset (e.g., the pattern of recent defaults). However, the modeler or other entities cannot easily determine from an analysis of the complicated ML model that the model will make this error in the analysis and, therefore, cannot easily limit the output of the complicated ML model by imposing standard regularization penalties.

The simple and interpretable ML models also find application in various industries because modeling data (for example, data used to train and test the ML model) may not always match data inputs applied to the ML models in the future. A fundamental assumption of ML modeling is that the corresponding systems, methods, and modelers train and test the ML models using data that is generally from a similar underlying distribution as the data on which the ML models are to be applied. This assumption, however, may be improper in certain fields, such as the financial field. Accordingly, the ML models applied in the financial field (and similar fields) should be available for inspection and analysis regarding whether the ML models have picked up patterns that may only apply to historical data or otherwise mischaracterize future data.

The systems and methods described herein implement scorecard models that encapsulate state-of-the-art techniques for binning, regularization, and optimization, among others. In particular, the systems and methods described herein implement scorecard tools that, according to at least some embodiments: (1) incorporate novel constraints used in corresponding modeling; (2) perform binarsity regularization that promotes model simplicity; (3) utilize efficient techniques to optimize the model parameters; and (4) perform a novel type of supervised binning by training a boosted decision tree model with stumps.

The scorecard tools are based on scorecard type ML models that are easily inspected and interpreted to verify outputs and functionality of the ML models. The disclosure below provides approaches to train the scorecard models, useful techniques to increase training speeds, and considerations for constraining behavior of the scorecard models.

Additionally, the systems and methods described herein introduce various constraints that may be applied to the scorecard tools described herein, such as protected bins and/or zero-base instances. Additionally, the systems and methods may introduce techniques to efficiently optimize ML model parameters. Furthermore, the systems and methods herein described a new binning scheme, whereby bin cuts for each variable of the binning scheme are determined by constructing a boosted decision tree model consisting of a large number of stumps, which can also serve to build a scorecard model. The disclosure below also illustrates efficiencies and benefits of introducing a binarsity penalty to ML model outputs as compared to standard penalties (for example, a standard L1 penalty) or utilizing supervised binning.

To facilitate an understanding of the systems and methods discussed herein, a number of terms are described below. The terms described below, as well as other terms used herein, should be construed to include the provided descriptions, the ordinary and customary meaning of the terms, and/or any other implied meaning for the respective terms. Thus, the descriptions below do not limit the meaning of these terms, but only provide exemplary definitions.

Data Store: Includes any computer readable storage medium and/or device (or collection of data storage mediums and/or devices). Examples of data stores include, but are not limited to, optical disks (for example, CD-ROM, DVD-ROM, and so forth), magnetic disks (for example, hard disks, floppy disks, and so forth), memory circuits (for example, solid state drives, random-access memory (“RAM”), and so forth), and/or the like. Another example of a data store is a hosted storage environment that includes a collection of physical data storage devices that may be remotely accessible and may be rapidly provisioned as needed (commonly referred to as “cloud” storage).

Database: Includes any data structure (and/or combinations of multiple data structures) for storing and/or organizing data, including, but not limited to, relational databases (for example, Oracle databases, MySQL databases, and so forth), non-relational databases (for example, NoSQL databases, and so forth), in-memory databases, spreadsheets, as comma separated values (“CSV”) files, extendible markup language (“XML”) files, TeXT (“TXT”) files, flat files, spreadsheet files, and/or any other widely used or proprietary format for data storage. Databases are typically stored in one or more data stores. Accordingly, each database referred to herein (for example, in the description herein and/or the figures of the present application) is to be understood as being stored in one or more data stores.

Database Record and/or Record: Includes one or more related data items stored in a database. The one or more related data items making up a record may be related in the database by a common key value and/or common index value, for example.

Output, Event Notification, Notification, and/or Alert: Includes electronic any notification sent from one computer system to one or more other computing systems. For example, a notification may indicate a new record set or changes to one or more records of interest. Notifications may include information regarding the record change of interest, and may indicate, for example, to a user, an updated view of the data records. Notifications may be transmitted electronically, and may cause activation of one or more processes, as described herein.

Historical data (also referred to as event data) may generally refer, in some embodiments, to data associated with any event associated with the individual, such as historical financial data, medical data, and so forth. The historical data may include transaction data, historical records, and so forth for various individuals. The transaction data may also include non-financial exchanges, such as login activity, Internet search history, Internet browsing history, posts to a social media platform, medical history, criminal history, or other interactions. In some implementations, the users may be machines interacting with each other (e.g., machine-to-machine communications). Transaction data may be presented in raw form. Raw transaction data generally refers to transaction data as received by the transaction processing system from a third-party transaction data provider. Transaction data may be compressed. Compressed transaction data may refer to transaction data that may be stored and/or transmitted using fewer resources than when in raw form. Compressed transaction data need not be “uncompressible.” Compressed transaction data preferably retains certain identifying characteristics of the user associated with the transaction data such as behavior patterns (e.g., spend patterns), data cluster affinity, or the like.

Entity: depending on the context, may refer to a person, such as an individual, consumer, or customer, and/or may refer to an entity that provides input to the system and/or an entity that utilizes a device to receive the event notification, notification or alert (for example, a user who is interested in receiving notifications upon the occurrence of the newly generated record set or changes to records of interest). Thus, in the first context, the terms “user,” “individual,” “consumer,” “business person,” and “customer” should be interpreted to include single persons, as well as groups of users, such as, for example, married couples or domestic partners, organizations, groups, and business entities. Additionally, the terms may be used interchangeably. In some embodiments, the terms refer to a computing device of a user rather than, or in addition to, an actual human operator of the computing device.

An entity may generally refer to one party involved in a transaction. In some implementations, an entity may be a merchant or other provider of goods or services to one or more users, a financial institution, a bank, a credit card company, a hospital, a local government, an individual, a lender, or a company or organization of some other type.

A model may generally refer to a machine learning construct which may be used by a computing system to automatically generate a result or outcome. A model may be trained. Training a model generally refers to an automated machine learning process to generate the model that accepts an input and provides a result or outcome as an output. A model may be represented as a data structure that identifies, for a given value, one or more correlated values. For example, a data structure may include data indicating one or more categories. In such implementations, the model may be indexed to provide efficient look up and retrieval of category values. In other embodiments, a model may be developed based on statistical or mathematical properties and/or definitions implemented in executable code without necessarily employing machine learning.

Embodiments of the present disclosure may enable various entities (for example, hospital, financial institutions, local government, schools, and the like) to determine risk assessment scores for an individual based on historical references for the individual. The embodiments apply various machine learning (ML) and/or artificial intelligence (AI) algorithms to data stores that include, among other information, historical records for the individual and historical records for other individuals with similar histories. The embodiments may generate risk assessment scores and/or analyses for the individual based on the historical information.

shows a network diagram of various components that communicate via a communication network(hereinafter “network”), such as the Internet, and form a risk assessment platformthat generates, trains, and applies ML models to records and/or information available to and/or processed by the risk assessment platform. The platformcomprises the network, a dynamic ML modeling system, a computing device, a first data store, a second data store, and external computing device(s). The networkincludes communication links shown enabling communication among the components of the platform.

The computing deviceis shown communicatively coupled to the dynamic modeling systemvia an optional localized manner (for example, via an optional local communication link, the optional nature indicated by a dashed communication link between the dynamic modeling systemand the computing device) and in an external manner where communications occur through the network. In some embodiments, the dynamic modeling systemis integrated into the computing deviceor vice versa. Furthermore, in some embodiments, one or more of the first data storeand the second data storeare combined into a single data store that is local to the computing deviceor remote from the computing device(not shown). In some embodiments, two or more of the components shown inabove are integrated, one or more components are excluded, or one or more components not shown inare added to the platform. The platformmay be used to implement systems and methods described herein.

In some embodiments, the networkmay comprise any wired or wireless communication network by which data and/or information may be communicated between multiple electronic and/or computing devices. The networkmay be used to interconnect nearby devices or systems together, employing widely used networking protocols. The various aspects described herein may apply to any communication standard, such as a wireless 802.11 protocol. The computing devicemay comprise any computing device configured to transmit and receive data and information via the networkfor an entity or in response to a request from the entity. The entity may be a financial institution such as a business, bank, credit card company, a non-profit organization, an educational institution, a healthcare provider, an insurer, and any other unit that can utilize risk assessment information or generate risk assessment information. In some embodiments, the request is a request to compute a risk score or similar score for an identified individual. The computing devicemay include or have access to one or more databases (for example, the first data storeand the second data store) that include various records and information that may be used to generate attributes, determinations, and/or scores for risk assessment for an individual, company, and so forth and generate outputs corresponding thereto. These attributes (and corresponding information) may be used to dynamically generate ML models, for example scorecard ML models, used to generate risk assessment scores and associated information for use in risk prediction problems.

The computing devices,may comprise any computing device configured to transmit and receive data and information via the networkfor application by the dynamic modeling system, which generates, trains, and applies ML models to the data in response to received requests. In some embodiments, the computing devicerepresents a centralized computing device that performs at least a portion of the processing described herein. For example, the computing deviceimplements the scorecard ML models described herein to generate probability scores or other results. The computing devicemay further analyze records stored in the first data storeand/or the second data store(or provided with the score request) and generate an output regarding the particular entities or identities and/or perform one or more actions based on the performed analysis and/or the transmitted and received data and information. In some embodiments, the computing devicesrepresent a customer or user device that the customer or user utilizes to access the platformand submit a request and/or data based on which the platformapplies the scorecard model(s) to determine a risk assessment score for an identified individual.

In some embodiments, the one or more computing devices,comprise mobile or stationary computing devices. In some embodiments, the computing devicesprovide users with remote access to the networkand the platform.

The first data storemay comprise one or more databases or data stores and may also store data regarding any identities (for example, name information, address information, activity information, and so forth). Using an example use case, the first data storecomprises credit score data that includes name information, address information, contact information, financial information, as well as other credit related data. In some embodiments, the credit score database may provide data for individuals or entities within particular geographic areas or for the entire platform.

The second data storemay also comprise one or more databases or data stores from a different source as compared to the first data store. The second data storemay also store data regarding corresponding to entities or identities, for example relationship data, transaction data, behavioral data, and so forth. In the example use case, the second data storecomprises one or more of property rental or ownership information, membership data, marketing data, public records data, business information, eCommerce data, digital browsing or footprint data, location data, and so forth. This data may be organized based on or according to identifiers common between the first data storeand the second data store, and so forth. In some embodiments, one or more of the first data storeand the second data storecomprise data from publicly available and/or private sources.

The dynamic modeling systemmay process data from the first data storeand the second data storeand also dynamically generate and/or apply one or more artificial intelligence models based on requests or inputs provided by users via the computing devices. The dynamic modeling systemmay dynamically apply one or more machine learning models to data obtained from one or more of the first data store, the second data store, and/or the users (via the one or more computing devices). In some embodiments, the machine learning models may be generated or adapted dynamically by the dynamic modeling systemas the inputs and data change. For example, the dynamic modeling systemmay generate changing ML (or other artificial intelligence) models in real-time based on the inputs received from the user (for example, a particular identifier, and so forth). In some embodiments, the generated models themselves may be dynamically applied to the inputs and data. For example, the ML models generated by the dynamic modeling systemmay create various score, metrics, and/or data points based on the data sourced from the first and second data storesand, respectively, and the users (for example, filters, threshold criteria, and so forth). In some embodiments, the dynamic modeling systemmay automatically adjust the machine learning models to meet pre-selected levels of accuracy and/or efficiency.

In some embodiments, the dynamic modeling systemadapts to data from the first data store, the second data store, or from users that is constantly changing. For example, the data in the first data storeand the second data storeis constantly updated and is different for each analysis of an identifier or entity. Using the example use case, a first user associated with a first financial institution may be interested in determining whether a first individual is likely to default on a loan based on the first individual's historical financial data in view of a first set of requirements, which is stored in one of the first data storeand the second data store. A second user from a second financial institution may be interested in determining whether the first individual is likely to default based on a second set of requirements or a different set of data (for example, just the historical data provided in the request). Accordingly, the data to which the ML models are applied by the dynamic modeling systemwill likely be constantly changing in some embodiments. Thus, the processing and/or model generation performed by the platformmay change for each user, each request, each individual, and so forth. Additionally, the data obtained from the first and second data storesandwill likely change over time as records in the data stores are updated, replaced, and/or deleted. Accordingly, the dynamic modeling systemmay dynamically generate and/or apply machine learning models to handle constantly changing data and requests.

Based on the user requests, as will be detailed herein, the requests may be filtered to eliminate those requests that need not be completed. For example, in the example use case, requests may be filtered to eliminate those requests that are predetermined to have particular risk assessment scores (for example, previously determined scores that were determined within a specified threshold of time). Accordingly, the dynamic modeling systemmay reduce unnecessary data processing by excluding certain requests.

In various embodiments, large amounts of data are automatically and dynamically processed interactively in response to user inputs, and the calculated data is efficiently and compactly presented to a user by the platform. Thus, in some embodiments, the data processing, machine learning, and generating of outputs and/or user interfaces described herein are more efficient as compared to previous data processing and output generation.

The various machine learning models and processing of data to identify risk assessment scores and comparisons, dynamic data processing, and output generation of the present disclosure are the result of significant research, development, improvement, iteration, and testing. This non-trivial development has resulted in the machine learning models and output generation described herein, which may provide significant efficiencies, improvements in accuracy and consistency, and advantages over previous systems.

Various embodiments of the present disclosure provide improvements to various technologies and technological fields. For example, as described above, existing data storage and processing technology (including, for example, in memory databases) is limited in various ways (for example, manual data review is slow, costly, and less detailed; data is too voluminous; and so forth, and existing ML models are too abstract and not explainable and can result in improper or incorrect outputs when applied to the data in the first and second data storesand), and various embodiments of the disclosure provide significant improvements over such technology. Additionally, various embodiments of the present disclosure are inextricably tied to computer technology. In particular, various embodiments rely on application of machine learning models, acquisition and processing of data, and presentation of output information via interactive graphical user interfaces or reports. Such features and others (for example, processing and analysis of large amounts of electronic data) are intimately tied to, and enabled by, computer technology, and would not exist except for computer technology. For example, the interactions with data sources and displayed data described below in reference to various embodiments cannot reasonably be performed by humans alone, without the computer technology upon which they are implemented. Further, the implementation of the various embodiments of the present disclosure via computer technology enables many of the advantages described herein, including more efficient interaction with, more accurate and consistent processing of, and presentation of, various types of electronic data. Furthermore, the scorecard models described herein may improve model governance, implementation, accountability, and performance. In some instances, the systems and methods described herein may be employed in marketing, insurance underwriting, fraud detection, and similar use cases for individuals and/or businesses.

Dynamic Modeling System

is a block diagram corresponding to an aspect of a hardware and/or software component of an example embodiment of a devicein the systemof. The hardware and/or software components, as discussed below with reference to the block diagram of the devicemay be included in any of the components of the system(for example, the dynamic modeling system, the computing device, the external computing devices, and so forth). These various depicted components may be used to implement the systems and methods described herein.

In some embodiments, certain modules described below, such as a user interface module, a ML/modeling module, or a report moduleincluded with the dynamic modeling systemmay be included with, performed by, or distributed among different and/or multiple devices of the system. For example, certain user interface functionality described herein may be performed by the user interface moduleof various devices such as the computing device.

In some embodiments, the various modules described herein may be implemented by either hardware or software. In an embodiment, various software modules included in the devicemay be stored on a component of the deviceitself (for example, a local memoryor a mass storage device), or on computer readable storage media or other component separate from the deviceand in communication with the devicevia the networkor other appropriate means.

The devicemay comprise, for example, a computer that is IBM, Macintosh, or Linux/Unix compatible or a server or workstation or a mobile computing device operating on any corresponding operating system. In some embodiments, the deviceinterfaces with a smart phone, a personal digital assistant, a kiosk, a tablet, a smart watch, a car console, or a media player. In some embodiments, the devicemay comprise more than one of these devices. In some embodiments, the deviceincludes one or more central processing units (“CPUs” or processors), I/O interfaces and devices, memory, the mass storage device, a multimedia device, the user interface module, the modeling module, the report module, and a bus.

The CPUmay control operation of the dynamic modeling system. The CPUmay also be referred to as a processor. The processormay comprise or be a component of a processing system implemented with one or more processors. The one or more processors may be implemented with any combination of general-purpose microprocessors, microcontrollers, digital signal processors (“DSPs”), field programmable gate array (“FPGAs”), programmable logic devices (“PLDs”), controllers, state machines, gated logic, discrete hardware components, dedicated hardware finite state machines, or any other suitable entities that can perform calculations or other manipulations of information. In some embodiments, the CPUmay perform one or more of the steps or processes described herein, for example a computing step, training of the ML models, or application of the ML models, and so forth.

The I/O interfacemay comprise a keypad, a microphone, a touchpad, a speaker, and/or a display, or any other commonly available input/output (“I/O”) devices and interfaces. The I/O interfacemay include any element or component that conveys information to the user of the device(for example, a requesting financial entity, a hospital, a local government, educators, or any other entity) and/or receives input from the user. In one embodiment, the I/O interfaceincludes one or more display devices, such as a monitor, that allows the visual presentation of data to the user. More particularly, the display device provides for the presentation of GUIs, application software data, websites, web apps, and multimedia presentations, for example.

In some embodiments, the I/O interfacemay provide a communication interface to various external devices. For example, the dynamic modeling systemand/or the consumer devicesare electronically coupled to the network(), which comprises one or more of a LAN, WAN, and/or the Internet. Accordingly, the I/O interfaceincludes an interface allowing for communication with the network, for example, via a wired communication port, a wireless communication port, or combination thereof. The networkmay allow various computing devices and/or other electronic devices to communicate with each other via wired or wireless communication links.

The memory, which includes one or both of read-only memory (ROM) and random-access memory (“RAM”), may provide instructions and data to the processor. For example, data received via inputs received by one or more components of the dynamic modeling systemand/or the consumer devicesmay be stored in the memory. A portion of the memorymay also include non-volatile random-access memory (“NVRAM”). The processortypically performs logical and arithmetic operations based on program instructions stored within the memoryand program instructions received from other sources. The instructions in the memorymay be executable to implement the methods described herein. In some embodiments, the memorymay be configured as a database and may store information that is received via the user interface moduleor the I/O interfaces and devices.

The devicealso includes the mass storage devicefor storing software, executable instructions or information (for example, the generated models or data obtained to which the models are applied), and so forth. Software shall be construed broadly to mean any type of instructions, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Instructions may include code (for example, in source code format, binary code format, executable code format, or any other suitable format of code). The instructions, when executed by the one or more processors, cause the processing system to perform the various functions described herein. Accordingly, the dynamic modeling systemand/or the consumer devicesmay include, for example, hardware, firmware, and software, or any combination therein. The mass storage devicemay comprise a hard drive, diskette, solid state drive, or optical media storage device. In some embodiments, the mass storage device may be structured such that the data stored therein is easily manipulated and parsed.

As shown in, the dynamic modeling systemincludes the modeling module. As described herein, the modeling modulemay dynamically generate one or more models for data obtained from the data stores or the user. In some embodiments, the modeling modulemay also apply the generated models to the data. In some embodiments, the one or more models may be stored in the mass storage deviceor the memory. In some embodiments, the modeling modulemay be stored in the mass storage deviceor the memoryas executable software code that is executed by the processor. This, and other modules in the dynamic modeling system, may include components, such as hardware and/or software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. In the embodiment shown in, the dynamic modeling systemis configured to execute the modeling moduleto perform the various methods and/or processes as described herein.

In some embodiments, the report modulemay be configured to generate a report, notification, or output mentioned and further described herein. In some embodiments, the report modulemay utilize information received from the dynamic modeling system, the data acquired from the data stores, and/or the user of the consumer deviceto generate the report, notification, or output for a specific entity (for example, a client or customer) looking to establish a relationship with the individual for whom the risk assessment is requested. For example, the dynamic modeling systemmay receive information from a client entity via the networkthat the dynamic modeling systemuses to acquire information from the first and second data storesand, respectively and generate and apply models for processing of the information from the client and/or from the data stores and identify corresponding risk assessment scores or values or information. In some embodiments, the generated report, notification, or output may comprise a data file including the risk assessment score and/or factors contributing to the risk assessment score. In some embodiments, the generated report, notification, or output does not include any personal identifying information (PII) of the consumers whose risk assessment score (or ranking or similar value) was generated. In some embodiments, the report modulemay include information received from the client in the generated report, notification, or output.