Machine learning system for forecasting customer demand

Forecasting customer demand is a cross-functional challenge that is fraught with difficulty. For example, supply chain management and marketing organizations often have issues correctly forecasting customer demand, which can result in inefficiency, incorrect financial plans, shortages, and other business problems. Customer demand is informed by a number of complex factors related to one another in complex ways. For example, on occasion customer demands lag customer actions. Machine learning (ML) models generated or created vis-à-vis an ML algorithm can be trained to forecast customer demand. These ML models can be trained with sampled data (e.g., training data) to make predictions or decisions without being explicitly programmed to do so. However, generating prediction models to accurately forecast future customer demand is difficult, even with ML algorithms. Prediction models often overfit and underfit training data resulting in predictions that may lack the desired accuracy. In the business context, ML challenges, such as overfitting and underfitting, will cause inaccurate predictions of customer demand, which can create significant supply chain problems, among other business challenges.

The figures are not exhaustive and do not limit the present disclosure to the precise form disclosed.

One of the primary objectives of various business functions, including the marketing function, is to determine customer demand. Most conventional supply chain management/marketing dashboards measure and report customer demand as lagging indicators of performance. However, this lag makes customer demand less desirable as a predictive indicator. Lag is the period of time between customer intent and customer demand. Customer intent is one or more actions a customer takes target digital marketplace that indicate the customer is considering making a purchase. Customer intent, however, is not and does not guarantee an actual purchase, but may or may not lead to an actual purchase. In some embodiments, such customer intent may be digital intent, which is one or more digital actions that can be observed/monitored (e.g. email views, clicks on a link, navigation to a webpage), and that may or may not lead to purchases on a target digital marketplace (e.g. website, application, program). In one example, the customer digital intent data is captured using an analytics platform/tool that captures such intent data from a target digital marketplace (or from multiple target digital marketplaces). Customer demand data can be, for example, extracted from a customer relations management (CRM) system. CRM systems track customer relationships with the target digital marketplace from a sales lead to a completed sale. In one configuration, customer demand is based on lead volumes that are entered into the CRM system. The lead volumes are converted into customer demand upon quantifiable sales (e.g., when sales are accepted by a sales team).

Customer demand is present or foreseeable customer orders or purchases. Customer demand can be characterized by customer demand data, such as records of orders or purchases. While customer demand could be analyzed in a time-series manner to predict future demand based on patterns in past demand, the value of such predictions rely heavily on the accuracy and continued applicability of patterns found in past customer demand data. Such prediction mechanisms struggle to capture changes in demand patterns, anomalous changes in demand, and latent thresholds in demand not captured by the modeled patterns.

Customer intent data, while not perfectly correlated with future demand, can indicate future demand, albeit with lag. The lag can include any amount of time. For example, the lag could be daily, weekly, quarterly, or fractions or multiples of any other period of time. Lag may not be uniform across different products, companies, or customers, or over time.

Often, supply chain management/marketing dashboards use lead volumes and opportunity pipelines to solely measure and report customer demand. These supply chain management/marketing dashboards, lack the ability to accurately predict/forecast customer demand due in part to the fact that, customer demand can lag customer intent. For example, product sales can occasionally lag customer actions on a target digital marketplace if a customer engages with a webpage but does not actually make a purchase from the webpage until some time passes. Although some ML algorithms have attempted to forecast customer demand, forecasting lagged customer demand is extremely difficult, since ML algorithms tend to overfit or underfit customer demand. For example, sometimes a temporary surge in customer intent can result in a ML algorithm forecasting a significant surge in demand. Without determining the relationship between customer intent and customer demand, ML algorithms can struggle with accurately forecasting periods of customer demand.

In an example system consistent with the teachings of this disclosure, both customer digital intent data and customer demand data are used to train and test prediction models to determine a relationship between the customer intent and customer demand. Here, sets of customer digital intent data and sets of customer demand data are used by the system to generate a plurality of prediction models. The system proposed herein can determine the relationship between customer intent and customer demand by generating multiple prediction models using sets of customer digital intent data and customer demand data. The relationship between the customer intent and customer demand can be used by supply chain management and marketing operations to increase the lead time to adjust the supply of products.

Each prediction model attempts to predict whether a relationship holds true between the customer digital intent data and the customer demand data. For example, a first prediction model may determine that the most likely lag between customer digital intent data and customer demand data for a specific product is 3 weeks. Each prediction model is trained using a training dataset comprising customer digital intent data and customer demand data, and validated using a testing dataset comprising customer digital intent data and customer demand data. In one configuration, a first training dataset and first testing dataset is used to train and validate each prediction model. In another configuration, a first training dataset and a second testing dataset is used for each model (e.g., each prediction model is trained and validated using a unique training dataset and testing dataset). Customer digital intent data used to generate training data and testing data includes customer high value actions. A “High value action” is defined as any action performed by the customer on a user interface (UI) that will likely result in a sale. The term “HVA data” can refer to captured data that is representative of the high value actions performed by the customer on the UI that will likely result in the sale. In order for an action to be considered/characterized as likely to result in a sale, that action should be similar to or the same as a previous customer action that results in an actual sale.

The system determines an amount of customer demand data to be used to generate training and testing datasets (the amount of customer demand data used to generate testing and training datasets is sometimes referred to as pipeline data). Each dataset includes weekly lagged HVA data (e.g., HVA data at 2 weeks, HVA data at 4 weeks, etc.). The one or more datasets of HVA data used as training data are hereafter referred to as “selected data.” The one or more sets of HVA data used as testing data is hereafter referred to as “non-selected data.”

Each prediction model is validated by a validator to determine an optimal prediction model. The prediction model that most accurately predicts the relationship between the customer digital intent data and the customer demand data is the optimal (i.e., most accurate) prediction model. During testing, the validator applies a testing dataset comprising non-selected sets of customer digital intent data and customer demand data to each prediction model of the plurality of the prediction models to determine the optimal predication model. It should be noted that the terms “optimize,” “optimal” and the like as used herein can be used to mean making or achieving performance as effective or perfect as possible. However, as one of ordinary skill in the art will recognize, perfection cannot always be achieved. Accordingly, these terms can also encompass making or achieving performance as good or effective as possible or practical under the given circumstances, or making or achieving performance better than that which can be achieved with other settings or parameters. The degree of optimization can also be varied in accordance with user needs/desires. In the context of the present disclosure an optimal prediction model can refer to a prediction model that most accurately predicts the testing dataset when compared to the plurality of prediction models.

Non-optimal prediction models can be re-trained by the modeling engine using feedback data. For example, data regarding the non-optimal prediction models can be used by the modeling engine to continuously re-train the ML prediction models to generate more accurate prediction models.

The optimal model is used to generate a forecast index that can be used in a dynamic executive dashboard to display the most likely lag between the customer digital intent and the customer demand for specific products. For example, using the forecast index, a user (e.g., a business executive) can easily determine that demand for a product will surge in about 3 weeks. In one example, the forecast index is a chart comprising a plurality of data points that forecast customer activity (e.g., a chart of points plotted over time or volume). The plurality of data points can include a number of customer digital actions, coverage ratios, and average deal size per opportunity. Based on the forecast index, business executives can adjust the development or supply of a product.

illustrates an example architecture for generating a forecasting index. Customer digital intent dataand customer demand dataare captured/generated at the input layer. Customer digital intent datais captured/generated by an analytics tool from customer digital interactions on a target digital marketplace. In one example the target digital marketplace is HPE.com©. In one example, the customer digital intent datais generated/using Adobe Analytics©. Customer demand data is extracted from a CRM system. The customer demand datacan be extracted from the CRM system using a variety of extraction techniques. In one example, the customer demand data is extracted using an extraction tool.

The customer digital intent dataincludes HVAs. In one configuration HVAs include: unique visitors, visits, assert response starts, asset response completes, click to call, sales chart click, click to sales agent, sales agent form complete, reactive chat click, reactive chat connected, product catalog product view, how to buy, gated asset downloads, un-gates asset downloads, proactive chat click, proactive chat connected, video views, click to social sharing, site searches, cart additions, cart views, carts, store-submit quote, checkouts, product detail page view, product list page view, store-configuration start, store-configuration complete, store-category/subcategory view, site search result clicks, store-quote new, sign in complete, account registration starts, store-product compare page view. In one configuration, the HVAs include: asset metrics comprising: gated asset downloads, un-gated asset downloads, video views, asset response starts, and asset response completes; chat metrics comprising: reactive chat click, proactive chat click, sales chat click, reactive chat connected, and proactive chat connected; call metrics comprising: click to call, and click to sales agent; cart metrics comprising: cart additions, cart views, and carts; product related metrics comprising: product detail page view, product list page view, product catalog product view, and store-product compare page view; purchase intent metrics comprising: how to buy, store-submit quote and checkouts; and account metrics comprising: sign in to complete and account registration starts. In one configuration, the HVAs comprise one or more: number of unique visitors, number of visits, number of clicks to call, number of video views, and number of reactive and proactive chats.

The HVA data associated with HVAs is categorized, according to weekly timestamps (e.g., 6 weeks, 8 weeks, 10 weeks, and 12 weeks). The weekly timestamps of the HVA data are representative of the lag, in this case, weekly lag, used by the modeling engine in the modeling layer to predict the relationship between the customer digital intent dataand the customer demand data

Customer digital intent dataand customer demand dataare sent from the input layerto the modeling layer, where they are used to generate a plurality of prediction models. The customer digital intent dataand the customer demand dataare received at the data pre-processing enginewhere they are combined and lagged to according to various combinations. The data pre-processing engine uses customer digital intent data and customer demand data captured over a period of time (e.g., 2 weeks, 4 weeks, 6 weeks, 8 weeks) to generate the training dataset and testing dataset. The combination and lagging of data according to various time periods is hereafter referred to as “feature engineering.”

Feature engineering is a process of merging customer digital intent datawith customer demand dataat weekly levels (e.g., 2 weeks, 4 weeks, etc.). To determine the appropriate relationship between customer digital intent dataand the customer demand data, the system creates multiple combinations of lagged datasets at weekly levels. The multiple combinations include multiple lagged variables from HVAs (e.g., four different variables from each lagged HVAs). In one example, for each individual HVA, 4 additional data points are created. Thus, assuming the existence of 30 HVA variables, 120 different variables may be generated. By generating additional variables, the system is better able to determine data-points that are indicative of the projected lag. The weekly lag can impact the model (e.g., a four week sum can impact a model more than a three week sum or a two week sum can impact a model more than a 4 week sum).

The system can determine the most accurate lag by generating a plurality of prediction models. In one example, each model is validated using a “goodness of fit measure”, to determine which lagged dataset is the best fit or the best representation of the training data. The “goodness of fit measure” is a method of determining which prediction model most accurately predicts the relationship between the customer digital intent data and the customer demand data.

The pre-processing engine further generates training datasetsA and testing datasetsB based on the customer digital intent dataand customer demand data. To prevent overfitting and bias, the combined customer demand data and customer digital intent data is split into two datasets: (i) a training dataset and (ii) a testing dataset. The training data includes selected data comprising data points chosen to represent a plurality of lagged high value actions. The remaining non-selected data points are used as testing data. For example, in a matrix comprising a plurality of data values, a first portion of data taken from a first portion of the matrix is used to create training data and a remaining portion taken from a second portion of the matrix (e.g., the unused portion) is used to create testing data.

The training dataA is sent to an ensemble engine, where it is rank ordered by the ensemble engineto generate rank order data. Because the training dataA includes a wide array of values captured over various times, rank ordering dataaids the plurality of ML algorithms in generating prediction models. For example, HVA data is assigned a relevance score that can be used to assess the relevance of the HVA data to the prediction model. In one embodiment, rank ordering further includes dimensionality reduction, which includes selecting the most relevant features, and ignoring irrelevant features when determining a feature rank order. Dimensionality reduction helps prevent overfitting by selecting and computing the most relevant HVA data. In one embodiment, combining the sets of models into the ensemble model includes determining a weighting of each model in the set of models based on a predictive power of the model. In one embodiment, the ensemble algorithm includes a plurality of ML algorithms comprising: a Recursive Feature Elimination (RFE) ML algorithm, a Boruta Feature Selection ML algorithm; and a Random Forest ML algorithm.

As seen further in, rank order datais sent from the ensemble engineto the modeling engineto generate a plurality of prediction models. The prediction model uses the different lags created by the preprocessing engine to determine the best lag for the customer digital intent data. The modeling engineincludes a plurality of ML modeling algorithms. The ML modeling algorithms may include, but are not limited to, a Stepwise Regression ML algorithm, a Gradient Boost (GBM) ML algorithm, an Extreme Gradient Boost (XGBoost) ML algorithm, and a Random Forest ML algorithm. Each ML modeling algorithm generates a respective prediction model. For example, a first model may be generated using the Stepwise Regression ML algorithm, a second model may be generated using the GBM ML algorithm, a third model may be generated using the XGBoost ML algorithm, and a fourth model may be generated using the Random Forest algorithm. In one embodiment, each prediction model is generated by a combination of ML modeling algorithms. For example, in one configuration, more than one ML algorithm can be used to generate a prediction model (e.g., a stepwise regression algorithm and a random forest algorithm can be used to generate a prediction model).

The plurality of prediction modelscomprise datasets representative of a computing system's (such as modeling engine's) interpretation of training dataA, including the previously described data flow. A family of feature sets within each dataset can be selected using, for example, iterative feature addition, until no features contribute to the accuracy of the models beyond a particular threshold. To improve the overall model, certain features can be removed from the set of features and additional models can be trained on the remaining features.

Each prediction model's prediction is tested (i.e., validated) by a validatorto determine the accuracy of the prediction model to forecast customer demand using the training dataA. Each prediction model from the plurality of prediction models is tested by comparing each prediction with testing dataB. The validatorcross validates each modeling algorithm by applying a goodness of fit measurement (e.g., R-square, RSME, R-square SD, RSME SD, etc.) to each model to determine over-fitting of training data. Each prediction model is assigned a score to determine the accuracy of each prediction model. The validatorgenerates an overall model score by normalizing and generating an aggregated weighted goodness of fit index. The model with the highest overall model score is selected by the validatorand sent to the forecasting enginein the transformation layer. The model with the highest overall model score is the optimal prediction model. In addition, sensitivity analysis of each prediction model can be performed to determine which HVA datais the largest driver of a predicted outcome.

The testing dataB is used by the validatorto test (i.e., validate) each prediction model to determine the most accurate prediction model (i.e., the optimal prediction model). To determine the accuracy of each prediction model, each prediction model of the plurality of prediction modelsis tested at the validatorusing testing dataB. The output of each prediction modelis charted against the testing data to determine the accuracy of each prediction model. The prediction model that most accurately predicts the relationship between the customer digital intent and the customer demand is the optimal prediction model.

In one configuration, the optimal prediction modelis the model that has the highest overall ranking. Each model is ranked by the validator according to a plurality of ranking factors to generate an overall ranking score. In one configuration, the plurality of ranking factors include the goodness of fit factors. The goodness of fit factors include a R-square measurement, a root mean squared (RSME) measurement, a R-square SD measurement, and a RSME SD measurement. The prediction model with the highest overall ranking score (i.e., the model that most accurately predicted the testing data) is the optimal prediction model. The optimal prediction modelis sent to the transformation layer, where it is used to generate a forecast indexcomprising model digital actions.

Although not the preferred configuration, it is foreseeable that in one configuration where time is not required, the validatorcan use a cross-validation technique to determine the accuracy of each prediction model. Here, the cross-validation is a resampling technique where the pipeline data is split into two parts (e.g., training dataA, and a testing dataB). The training dataA is used to train the prediction models, and the testing dataB is used to validate the predictions. The testing dataB can be unseen testing dataB.

illustrates an example architecture for generating a prediction model. The prediction model construction block includes a plurality of instructions for generating the plurality of prediction models. In one example, the plurality of instructions include instructions for: (i) ML prediction model generation, (ii) deterministic pattern learning, and (iii) recursive learning.

In addition to using one or more of the plurality of ML modeling algorithms to generate the plurality of prediction models, the system can apply deterministic pattern learning and recursive learning to tune each prediction model. Deterministic pattern learning is used to factor in uncertain dynamic environments into ML modeling algorithms to decrease over/under fitting. Without deterministic pattern learning, each prediction model may overshoot a prediction of customer demand by factoring in a temporary surge in customer demand. In one configuration, deterministic pattern learning includes one or more algorithms configured to represent a seasonality correction factor. The seasonality factor can be applied to each prediction model to prevent overshooting and undershooting customer demand, by smoothing (e.g., adjusting/altering) the data based on certain demand cycles. Without smoothing the training data, the plurality of prediction modelsmay undershoot customer demand following the uptick in seasonal demand, by over compensating for the apparent decreased customer demand. By smoothing out the data, each ML modeling algorithm can more accurately predict customer demand.

To make sure that the prediction models maintain a desired level of prediction accuracy, the prediction models are monitored by a prediction model monitor. The prediction model monitorcontinuously monitors the efficacy of one or more models in a real-world environment (e.g., a production environment) by capturing data regarding the accuracy of the prediction model. The prediction model monitorsends the captured data to the modeling engine for recursive learning. The recursive learning uses the captured data to generate a normalized weighted rank based on a goodness-of-fit framework, which assists the processor in continuously selecting an appropriate prediction model and helps to reinforce learning, and guarantee stable performance.

is a method of model construction and validation. The method includes receiving pipeline datacomprising time specific HVA data, pre-processing the pipeline datadata to generate training dataA and testing dataB, generating a plurality of prediction modelsusing training dataA, determining the optimal prediction modelby comparing each model to testing dataB, generating a forecast indexbased on the optimal prediction model, and retraining the non-optimal prediction models using feedback data.

At activity, the methodincludes receiving customer digital intent data and customer demand data from the input layer. The customer digital intent datainclude HVAs. The customer digital intent dataand the customer demand dataare received by the pre-processing enginein the modeling layer.

At activity, the methodincludes pre-processing the pipeline datato generate training dataA and testing dataB. Here, the pre-processing enginegenerates training dataA, and testing dataB. The training dataA is used to generate the plurality of prediction models. The testing dataB is used to determine the accuracy each prediction model. The most accurate prediction model is the optimal prediction model.

The testing dataA is rank-ordered using the ensemble engine. The ensemble engineuses the training dataA to generate rank order data. As previously mentioned, the training dataA includes lagged HVA data. The time lagged HVA datais rank-ordered by the ensemble engineto generate rank order data. The rank-ordered datacan be used to train the plurality of prediction models. In one embodiment, the ensemble-based ML algorithms include a Recursive Feature Elimination (RFE) ML algorithm, a Boruta Feature Selection ML algorithm, and a Random Forest ML algorithm.

At activity, the methodincludes generating a plurality of prediction models using rank order data. The modeling enginereceives rank order data. The modeling engineincludes a plurality of ML algorithms. In one embodiment, the ML modeling algorithms include the stepwise regression ML algorithm, the GBM ML algorithm, the XGBoost algorithm, and the random forest ML algorithm. Each ML modeling algorithm generates a respective prediction model.

At activity, the methodincludes determining the optimal prediction modelby validating each model against testing data. The optimal prediction modelis determined by cross-validating each model using a plurality of “goodness-of-fit” measures. Examples of goodness-of-fit measures include R-square, RSME, R-square SD, and RSME SD. Each goodness-of-fit measure is normalized and stored in an aggregated goodness-of-fit index. Each goodness-of-fit measure can be assigned a score. Each score can be aggregated in the goodness-of-fit index to generate an overall model score.

If the prediction model is the optimal prediction model, then the method includes activity. If the prediction model is a non-optimal model, then the method includes activity. At activity, one or more processorsdetermine whether or not a prediction model is an optimal prediction modelbased on the overall score assigned to each model at activity. For example, the optimal prediction modelis a first prediction model with a first overall score that is higher than a second prediction model with a second overall score.

At activity, the methodincludes retraining the plurality of prediction modelsusing testing dataB. As seen in, non-optimal models can be re-trained (i.e., re-learned) with feedback datato increase their prediction accuracy. Each prediction model can alter its prediction based on the feedback data.

At activity, the methodincludes generating a forecast indexbased on the optimal prediction model. The optimal prediction modelis sent from modeling layerto the transformation layer, where it is received by the forecasting engine. The forecasting enginegenerates a forecast indexbased on the optimal prediction model. The forecast indexis an estimation of opportunity counts that can be generated due to the customer's digital intent data. The forecast index includes a plurality of data points that forecast customer activity. The plurality of data points can be plotted over time, or over volume. The plurality of data points can include a number of customer digital actions, coverage ratios, and average deal size per opportunity. The forecast index can be sent to a user dynamic dashboardto adjust product supply based on the forecasted customer demand. The forecast indexincludes modeled digital actionsthat include coverage ratios, and average deal size per opportunity. The forecast indexallowed marketing teams to prepare for customer demand. For example, in one embodiment, a marketing team can use the forecast indexto determine whether customer demand is increasing or decreasing. In another embodiment, a logistics team can use the forecast indexto ramp up or ramp down production of a product according to projected demand.

illustrate an example computing component that may be used to generate a prediction model. Computing componentmay be, for example, a server computer, a controller, or any other similar computing component capable of processing data. In the example implementation of, the computing componentincludes a hardware processor, and a machine-readable storage medium.

The hardware processormay be one or more central processing units (CPUs), semiconductor-based microprocessors, and/or other hardware devices suitable for retrieval and execution of instructions stored in machine-readable storage medium. The hardware processor may be hardware processor. Hardware processormay fetch, decode, and execute instructions, such as instructions-, to control processes or operations for implementing the dynamically modular and customizable computing systems. As an alternative or in addition to retrieving and executing instructions, hardware processormay include one or more electronic circuits that include electronic components for performing the functionality of one or more instructions, such as a field programmable gate array (FPGA), application specific integrated circuit (ASIC), or other electronic circuits.

A machine-readable storage medium, such as machine-readable storage medium, may be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. Thus, machine-readable storage mediummay be, for example, Random Access Memory (RAM), non-volatile RAM (NVRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), a storage device, an optical disc, and the like. In some embodiments, machine-readable storage mediummay be a non-transitory storage medium, where the term “non-transitory” does not encompass transitory propagating signals.

As seen in, the machine-readable storage mediummay be encoded with executable instructions, for example, blocks-comprising instructions to model digital actions and generate a forecast index. The forecasting enginegenerates a forecast indexbased on the optimal prediction model. The forecast indexis an estimation of future sales opportunities (i.e., future high value action data) that can be generated due to the customer's digital intent data. The forecast index includes a plurality of data points that forecast customer activity. The plurality of data points can be plotted over time, or over volume. The forecast indexincludes modeled digital actionsthat include coverage ratios, and average deal size per opportunity.

As seen in, the machine-readable storage mediummay be encoded with executable instructions, for example, blocks-comprising instructions to perform a Recursive Feature Elimination, a Boruta Feature Selection, and a variable importance calculation. The ensemble engineis programmed to use ensemble-based ML algorithms to rank order testing data comprising time specific HVA data. Features from the data (e.g., HVAs) can be selected from the testing data to train the prediction models. Ensemble learning is employed to use multiple trained models to obtain better predictive performance than could be obtained from any individual constituent trained model. Ensembles are capable of yielding better results when there is a significant diversity among the models. Here, the ensemble learning combines multiple hypotheses to form a better hypothesis. For example, in one embodiment, the ensemble engineincludes a Recursive Feature Elimination (RFE) algorithm, a Boruta Feature Selection algorithm, and a Random Forest algorithm. Ensembles are capable of yielding better results when there is a significant diversity among the models. Therefore, disparate datasets from the plurality of data sources are can be beneficial in providing diversity among the models the ensembles combine.

Blockincludes instructions to perform a Recursive Feature Elimination (RFE). RFE is a feature selection ML algorithm that selects a subset of the most relevant features of a dataset. In one embodiment, the feature selection ML algorithm is executed by hardware processorof the ensemble engine. Here, the RFE receives the training dataA, as an input, and outputs rank order data. The RFE algorithm systematically searches through the training dataA to generate rank order datafrom HVA data. In one example, rank order dataincludes HVA datarank ordered according to the time of the action. The RFE is complete once all testing data is rank ordered.

Blockincludes instructions to perform Boruta Feature Selection (BFS). BFS is a feature selection ML algorithm that works as a wrapper algorithm around a random forest ML algorithm. The Boruta Feature Selection algorithm is executed by hardware processorof the ensemble engine. Here, the BFS receives the training dataA, and outputs rank order data. Specifically, the BFS randomizes the training dataA by creating shuffled copies of all features (i.e., shadow features). The BFS algorithm trains a Random Forest classifier to recognize and classify customer digital intent data by applying a feature importance measure to evaluate the importance of each lagged HVA data. At every iteration, the Boruta algorithm checks whether each HVA data has a higher importance than the shadow feature. If the HVA data does not have a higher importance, it is rejected. If the HVA data has a higher importance it is confirmed. The BFS is complete once each HVA data is confirmed or rejected.

Blockincludes instructions to perform a variable importance calculation by a Random Forest ML algorithm (i.e., a Gini Importance algorithm). The Gini Importance ML algorithm is executed by hardware processorof the ensemble engine. The gini importance algorithm receives training dataA as an input, and outputs rank order dataas an output. Specifically, the Gini Importance algorithm receives the training dataA and feeds the pipeline datainto the Random Forest algorithm. However, unlike traditional Random Forest algorithms, the Gini Importance algorithm returns an plot of the important features (e.g., the time lagged HVA data). The Random Forest ML algorithm outputs rank-ordered data.

As seen in, the machine-readable storage mediummay be encoded with executable instructions, for example, blocks-comprising: instructions to generate a prediction model using a Stepwise Regression ML algorithm; instructions to generate a prediction model using a GBM ML algorithm; instructions to generate a prediction model using an XGBoost ML algorithm; and instructions to generate a prediction model using a Random Forest ML algorithm.

Blockincludes instructions to generate a prediction model using a Stepwise Regression ML algorithm. The Stepwise Regression ML algorithm is a step-by-step iterative construction of a regression model that involves selection of independent variables to be used in a final model. The Stepwise Regression ML algorithm is executed by hardware processorof the modeling engine. The Stepwise Regression ML algorithm receives rank order data, as an input and generates a prediction model as an output. The prediction model is sent to the validatorwhere a model performance validation operation is conducted on the prediction model to determine the accuracy of the prediction model. The Stepwise Regression ML algorithm performs a regression operation on rank order datato determine one or more values that can be used to generate a prediction model.

illustrates an example method of forecasting the relationship between customer intent and customer demand. The methodincludes receiving customer digital intent dataand customer demand dataat a pre-processing engine. The customer digital intent dataincludes time-lagged HVA data. The customer digital intent dataand the customer demand dataare used by the pre-processing engine to generate training dataA, and testing dataB. The ensemble engineuses the training dataA to generate rank order data. The rank order datais used by the modeling engineto generate a plurality of prediction models. The rank order datacomprises a ranked order of the training dataA comprising time-specific customer HVAs. As explained in further detail below, the modeling engineuses rank order datato generate a plurality of prediction models. Each prediction model of the plurality of prediction modelsis generated using a unique ML algorithm. Once generated, each prediction model is sent to the validatorand assigned an overall model score. The overall model score is based on each model's ability to correctly predict testing dataB. The prediction model with the highest overall model score (i.e., the most accurate model) is the optimal prediction model. The optimal prediction modelis sent to the forecasting enginewhere it is used to generate a forecast index. The forecast indexcan be used to adjust the supply of one or more products based on the forecasted customer demand.

At activity, the methodincludes receiving customer digital intent dataand customer demand data. The customer digital intent dataincludes data regarding customer interactions on a target digital marketplace. Customer interactions include any customer actions that can result in a purchase of a product (e.g., clicking an icon of a product, or selecting an item of interest). In one example, the customer digital intent data is gathered at weekly increments over a two year period. Customer demand datais extracted from a CRM system.