Inventory characterization and identification system

In current inventory management system, inventory that is not initially received and reported accurately during the inbound processes (e.g., by the company's Receiving Department), it becomes nearly impossible to maintain an accurate system or record. This affects all aspects of logistics services, manufacturing processes, and the supply chain.

Various embodiments are depicted in the accompanying drawings for illustrative purposes and should in no way be interpreted as limiting the scope of the embodiments. Furthermore, various features of different disclosed embodiments can be combined to form additional embodiments, which are part of this disclosure.

For purposes of this disclosure, the terms “container” or “shipping container” are used interchangeably to broadly refer to any container used for shipping. By way of non-limiting example, a container can include one or more of a box, a crate, a bag, or an envelope. As another example, the container can include a vessel for smaller items or a vessel for freight shipping.

A packing slip—which often lists the weights, dimensions, SKUs (Stock Keeping Units), number of units, etc. in a shipment—can play an important role in terms of inventory management. To that end, when a packing slip is unavailable, the supply chain procedures carried out in a warehouse can become disjointed and inefficient. For example, warehouse members may resort to manually opening boxes to count or identify materials, which can cause significant delays, introduce inventory uncertainties, and increase operation costs, among other things.

To address these or other challenges, a container management system can be implemented to apply a rigorous and automated process to identify the contents of a container, without relying solely on information from a packing slip and without requiring the container to be opened and the contents manually reviewed. The container management system can analyze the container to determine container information, evaluate the container information against stored inventory data, and characterize an identity and/or number of items in the container based on the evaluation. Furthermore, the container management system can generate a confidence parameter indicating how closely the container information matches the stored inventory data for the predicted item. Based on the identity of the item and/or the confidence parameter, the container management system can cause an action, such as updating the stored inventory information to indicate the predicted item has been received/processed (e.g., when confidence is high) or outputting an instruction to manually open and review the contents of the shipping container (e.g., when confidence is low). By electronically identifying the contents of the container (in many cases, without requiring the container to be opened), the container management system advantageously improves efficiencies and accuracies in inventory management, as well as improves the usage of facilities and labor.

The container management system can implement a machine learning system to apply a rigorous and automated process to identify the contents of containers accurately and efficiently. The machine learning system can enable recognition and/or identification of tens, hundreds, thousands, or millions of features and/or feature values for tens, hundreds, thousands, or millions of observations, thereby increasing accuracy and consistency and reducing delay associated with the relative resources (e.g., computing or human) required to be allocated for tens, hundreds, or thousands of operators to manually identify the contents of containers using the features or feature values.

The container management system can create and implement a confidence generation policy to provide an objective authority to govern how and when to calculate the confidence parameter. In particular, the confidence generation policy can be structured by rules and best practices for inventory management. By generating the confidence parameter according to the confidence generation policy and recording the predicted identity in the inventory data when the confidence parameter satisfies a confidence threshold, the container management system can advantageously ensure a congruity between the actual inventory in the warehouse and the stored inventory data.

In light of the description herein, it will be understood that the embodiments disclosed herein substantially improve inventory management efficiency and accuracy. Specifically, the embodiments disclosed herein enable a container management system to improve the performance of a reading station by enabling automated analysis of containers using a set of scans to obtain real-time measurements and other information. The ability to determine the contents of the container using the real-time information accurately and efficiently enables the underlying systems to manage inventory more efficiently and accurately by reducing the number of containers with unidentified items, which improves the usage of facilities; and reducing the number of containers for manual inspection, which improves the usage of labor, reduces processing time, and increases accuracies.

Thus, the presently disclosed embodiments represent an improvement at least in inventory management. Moreover, the presently disclosed embodiments address technical problems inherent within inventory management, image processing, and database management. These technical problems are addressed by the various technical solutions described herein, including scanning of the container, obtaining shipping information and inventory data, evaluating real-time measurements and shipping information against the inventory data, determining a confidence parameter according to a confidence generation policy, etc. Thus, the present application represents a substantial improvement on existing inventory management systems in general.

It will be appreciated that the inventive concepts described herein may be applied in contexts other than warehouse and inventory management. For example, similar techniques to those described herein can be applied in grocery store management, warehouse management, retail store management, online marketplace management, hospital management, etc.

illustrates an embodiment of an inventory management environmentfor applying an automated process for identifying the contents of a shipping container. The environmentincludes a reading station, an inventory management system, a container management system, and a network. In the illustrated embodiment, the reading stationincludes a container analyzer, a label printer, and an enhanced reality (XR) system. Furthermore, the inventory management systemincludes an inventory managerand an inventory catalog. To simplify discussion and not to limit the present disclosure,illustrates only one reading station, container analyzer, label printer, XR system, inventory management system, inventory manager, inventory catalog, and container management system, though multiple may be used.

Any of the foregoing components or systems of the environmentmay communicate via the network. Although only one networkis illustrated, multiple distinct and/or distributed networksmay exist. The networkcan include any type of communication network. For example, the networkcan include one or more of a wide area network (WAN), a local area network (LAN), a cellular network, an ad hoc network, a satellite network, a wired network, a wireless network, and so forth. In some embodiments, the networkcan include the Internet.

Any of the foregoing components or systems of the environment, such as any combination of the reading station, container analyzer, the label printer, the XR system, the inventory management system, the inventory manager, the inventory catalog, or the container management systemmay be implemented using individual computing devices, processors, distributed processing systems, servers, isolated execution environments (e.g., virtual machines, containers, etc.), shared computing resources, or so on. Furthermore, any of the foregoing components or systems of the environmentmay host or execute one or more client applications, which may include a web browser, a mobile application, a background process that performs various operations with or without direct interaction from a user, or a “plug-in” or “extension” to another application, such as a web browser plug-in or extension.

The reading stationcan capture or other obtain container information relating to one or more containers to be processed. In some cases, a container to-be-processed is unopened and/or includes one or more unknown items therein. The container information can include, but is not limited to, a physical attribute of the container. For example, the container information can include one or more dimensions of the container (e.g., length, width, height), a weight of the container, a composition of the container (e.g., cardboard, plastic, paperboard, etc.), a type of container (e.g., a vessel for freight shipping, a box, a crate, a bag, an envelope, etc.), or whether the box is sealed (unopened) or unsealed (opened). Furthermore, the container information can include, but is not limited to, a physical attribute of an item within the container or a collective attribute of items within the container. For example, the container information can include one or more dimensions of the item(s), a shape of the item(s), a composition of the item(s) (e.g., metal, plastic, paper, resin, etc.), or a 2D or 3D model of the item. Furthermore, the container information can include, but is not limited to, logistics information relating to the container. For example, the container information can include a date of arrival (e.g., Saturday, September 17), a time of day of arrival (e.g., 1:07 PM), a carrier (e.g., FedEx), a number of overall containers or similar containers in the same delivery, etc. In some cases, the container information includes data usable to determine a physical attribute of the container, a physical attribute of the item, or logistics information. The container information may be processed (e.g., in real-time) and communicated to the container management system, the inventory management system, or the like.

In the illustrated example, the reading stationincludes a container analyzer, a label printer, and an XR system. It will be appreciated that the reading stationrepresents an example reading station and other examples may use fewer, additional, or different components or arrangements. For example, in some cases, the reading stationmay not include label printerand/or an XR system.

The container analyzercan process a container to obtain container information. The container analyzermay include one or more devices for obtaining container information. For example, container analyzercan include, among other things, an imaging device, an item-detection device, a weighing device, a user input device, etc.

An imaging device can be configured to obtain one or more images of a container, such as images from different perspectives of the container. As another example, the imaging device may be configured to obtain one or more images of container documentation (e.g., a packing slip, a shipping label, etc. The reading stationand/or another component (e.g., the container management system) can process the one or more images to determine container information (e.g., in real time). In some cases, the images may be processed using a trained machine learning system. The container analyzercan store all of the captured images. Alternatively, the container analyzermay store only certain selected ones of the captured images. For example, the container analyzermay process or pre-process the images in real time or near real time to identify which images, if any, includes satisfactory images of a container. In some cases, the imaging device can be implemented as a camera.

An item-detection device can be configured to obtain information relating to an item within the container. For example, the item-detection device can be configured to obtain data to determined edges, shapes, compositions (e.g., metal, non-metal), bulkiness, a skeleton, etc. of the item(s) with the container. The reading stationand/or another component (e.g., the container management system) can process the information to determine container information (e.g., in real time). In some cases, the information may be processed using a trained machine learning system. In some cases, the imaging device can be implemented as an infrared camera, an x-ray device, a sonic imager, a metal detector, an ultrasonic probe, or other type of imaging probe.

A weighing device can be configured to obtain weight information relating to the container or an item within the container. For example, the weighing device can obtain the weight of the container. As another example, the weighing device can obtain the weight of an item in the container, such as by utilizing known weights of packages supplies (e.g., weight of the box itself, weight of packing filler, etc.). The reading stationand/or another component (e.g., the container management system) can process the weight information to determine container information (e.g., in real time). In some cases, the weight information may be processed using a trained machine learning system. In some cases, the imaging device can be implemented as a scale, a force sensor, etc.

A user input device can be configured to obtain container information via user input and/or communication with a data store (e.g., the inventory catalog). For example, the user input device can allow an operator to enter known container information, such as container information obtained via measurements performed by the operator and/or container information obtained from container documentation. In some cases, the user input device obtains logistics information or other container information.

The reading stationmay include hardware and software components for establishing communications over the network. The reading stationmay have varied local computing resources such as central processing units and architectures, memory, mass storage, graphics processing units, communication network availability and bandwidth, and so forth. Further, the reading stationmay include any type of computing system. For example, the reading stationmay include any type of computing device(s), such as desktops, laptops, and wireless mobile devices (for example, smart phones, PDAs, tablets, or the like), to name a few. The implementation of the reading stationmay vary across embodiments. For example, in some cases, one or more components of the reading station(e.g., the container analyzer) may be implemented as a portable or handheld device. As another example, in some cases, one or more components of the reading stationmay be implemented as a fixed platform or a device that is fixed in a particular location.

Through the use of the reading station, a user can scan containers singularly or in bulk. For example, in some cases, the reading stationcan perform one action on all of the containers scanned or can scan containers individually and perform more granular actions. For example, a shipment may include ten similar (e.g., seemingly identical in size, shape) containers. In some cases, the number of similar containers can be part of the container information and may be used to filter the options of possible items within the containers.

As shown, the reading stationcan include a label printerfor printing documentation for a container. For example, the reading stationcan include a graphical user interface, an application programming interface, etc. that can send data to the label printerfor the creation of labels for containers processed by the reading station. In some scenarios, as part of the processing, the label printercan generate a label to be affixed to, included in, or otherwise associated with the container. The label printercan be used to print the new label. In some embodiments, the new label can include information that is useable to identify some or all of the container information. In addition or alternatively, the new label can include information that is usable to identify the predicted identifier for the item in the container. In some cases, the new label included information similar to a packing slip. In some embodiments, the label printercan be configured to print a bar code or QR code and/or print onto a label that already includes a bar code or QR code.

The reading stationcan include an XR system. The XR system can be implemented as a wearable system (also referred to herein as an augmented reality (AR) system) and can be configured to present two-dimensional (2D) or three-dimensional (3D) virtual images to a user. The images may be still images, frames of a video, or a video, in combination or the like. The XR system can include a wearable device that can present a VR, AR, or MR environment, alone or in combination, for user interaction. The XR system can be a head-mounted device (HMD) which is used interchangeably as an AR device (ARD).

The XR systemcan be configured to display a virtual model (e.g., a 2D model, a 3D model, an augmented reality (AR) model, a mixed reality (MR) model, or a digital image) of an item determined to be within a container. For example, the XR system may overlay or superimpose a virtual model of the item on a respective package, thereby providing an augmented reality environment. In some cases, the reading stationgenerates or constructs the virtual model of the item using data captured by the reading station. For example, the reading stationcan scan (e.g., using x-ray or other item-detection device) the container can generate a dataset of items. Definable features of the geometry of the item can be determined based on the scanned dataset. In this way, dimensions, shapes, contours, sizes, or the like of the item can be determined based on the data relating to the definable features from the scanned dataset, and the virtual model can be generated. In some cases, the XR systemis implemented as a head-mounted display, such as a see-through head-mounted display. For example, the XR systemcan render at least a portion of the virtual model as an augmented reality overlay that overlays on the container when the container is viewed through the see-through head-mounted display.

Although reference is made throughout the specification to the reading stationperforming various analytical or processing functions, it will be understood that, in some embodiments, the container management systemperforms these functions, and the reading stationis used to acquire data for communicating to the container management system. In such embodiments, the reading stationcan receive notifications of processing functions performed by the container management system, such as indications of the processing completion. Accordingly, in some embodiments, the amount of processing performed by the reading stationcan be reduced and/or minimized, and the reading stationcan act as a conduit to the container management system. In this way, the hardware requirements and costs of the reading stationcan be reduced in favor of a larger or more robust container management system. In certain embodiments, the reading stationcan include the container management systemand perform all of the functions described herein.

The inventory management systemcan include an inventory managerand an inventory catalog. As described herein, the inventory management systemmay be implemented using individual computing devices, processors, distributed processing systems, servers, isolated execution environments (for example, virtual machines, containers, etc.), shared computing resources, or so on.

The inventory managercan be used to manage, create, develop, or update inventory in the inventory management environment. For example, the inventory managercan maintain the inventory catalogwith inventory data. The inventory managercan populate the inventory catalogand/or update it over time with inventory data that receives and/or generates. As inventory data changes, the inventory managercan update the inventory catalog. In this way, the inventory catalogcan retain an up-to-date database of inventory data.

In some cases, the container management systemcan include a local data store (not shown) for storing inventory data. For example, the local data store can include inventory data corresponding to recently created, developed, or updated scans performed by the reading station. In some cases, the inventory managercan maintain the inventory catalogby pinging the container management systemfor inventory data or passively receiving it based on the container management systemindependently reporting the inventory data. For instance, the inventory managercan ping or receive information from the container management systemat predetermined intervals of time, such as every X number of hours, or every X days, etc. In addition or alternatively, the container management systemcan be configured to automatedly send its inventory data to the inventory managerand/or the inventory catalog. In some cases, the inventory catalogcan be manually updated or updated responsive to new inventory orders.

Furthermore, the inventory managercan transmit the inventory data to the container management systemvia data packets, such as part of an update to the container management system. For instance, the inventory managercan communicate inventory updates to the container management systemat predetermined intervals of time, such as every day, week, or month. In some cases, the inventory managerprovides the inventory data when the inventory data, or updates thereto, becomes available from the inventory manager. In some cases, the inventory managerprovides the inventory data when an order is submitted or received. As another example, the container management systemcan query the inventory managerfor inventory data or can download inventory data from the inventory catalog.

The inventory catalogcan store inventory data. In some embodiments, the inventory data can include container information, such as historical container information. In some cases, the inventory data can include information such as a product's name, SKU number, description, pricing, quantity, dimensions, weight, or other container information. In some cases, the inventory catalogincludes a comprehensive or semi-comprehensive, itemized list that details every product a company has in stock or has ordered, including raw materials, work-in-progress items, finished goods, etc. In some cases, inventory catalogincludes information regarding inventory purchases, inventory deliveries, items not yet received, or unexpectedly received items. In some cases, the inventory catalogincludes expected shipping information for some or all of the items. For example, the inventory catalogmay indicate a Product Y is often shipped by Carrier G and is often received on Tuesdays. As described herein, by evaluating container information obtained by the reading station, as well as inventory information stored in the inventory catalog, the container management systemcan characterize (e.g., determine) the identity of an item within a container.

The inventory catalogcan be maintained (for example, populated, updated, etc.) by the inventory manager. As mentioned, in some embodiments, the inventory managerand inventory catalogcan be separate or independent of the container management system. Alternatively, in some embodiments, the inventory managerand/or inventory catalogare part of the container management system. Furthermore, in some cases, the inventory catalogcan be separate from or included in, or part of, the inventory manager. As described herein, a particular item identifier can be associated with various other inventory data. The item identifiers can be implemented as alphanumeric identifiers or other identifiers that can be used to uniquely identify one item identifier from another item identifier stored in the inventory catalog. For example, each item identifier can correspond to a particular product, and the associated inventory data can include information relating to that product. In some such cases, as described herein, the item identifier can be used to identifier associated inventory data. Similarly, actual inventory data can be used to identifier an item identifier.

The inventory catalogcan include or be implemented as cloud storage, such as Amazon Simple Storage Service (S3), Elastic Block Storage (EBS) or CloudWatch, Google Cloud Storage, Microsoft Azure Storage, InfluxDB, etc. The inventory catalogcan be made up of one or more data stores storing data that has been received from one or more of the reading station, the container management system, or the inventory manager, or data that has been received directly into the inventory catalog. The inventory catalogcan be configured to provide high availability, highly resilient, low loss data storage. The inventory catalogcan include Amazon CloudWatch metrics. In some cases, to provide the high availability, highly resilient, low loss data storage, the inventory catalogcan store multiple copies of the data in the same and different geographic locations and across different types of data stores (for example, solid state, hard drive, tape, etc.). Further, as data is received at the inventory catalogit can be automatically replicated multiple times according to a replication factor to different data stores across the same and/or different geographic locations.

The container management systemcan be used to identify the contents of a package. As described herein, the container management systemcan communicate with the reading stationto obtain container information and can communicate with the inventory management systemto obtain inventory data. Using the container information and inventory data, the container management systemcan determine an identity of the item(s) in a particular container. For example, as described herein, in some cases, the container management systemcan implemented one or more machine learning methods to characterize the identity of the item(s) based on images of the container, container information, inventory data, etc. Furthermore, in some cases, the container management systemcan generate a confidence parameter indicating a confidence associated with the determined identity of the item. In this way, the inventory management systemcan improve the ability to process containers and accurately determine the identity of items therein.

is a diagram illustrating an example of training a machine learning modelin connection with the present disclosure. The machine learning model training described herein may be performed using a machine learning system. The machine learning system may include or may be included in a computing device, a server, a cloud computing environment, or the like, such as the container management systemof.

As shown by reference number, a machine learning model may be trained using a set of observations. The set of observations may be obtained and/or input from historical data, such as data gathered during one or more processes described herein. For example, the set of observations may include data gathered from the reading station, as described elsewhere herein. In some implementations, the machine learning system may receive the set of observations (e.g., as input) from the reading stationor from a storage device. In some cases, the set of observations may include data gathered from the inventory management system, as described elsewhere herein.

As shown by reference number, a feature set may be derived from the set of observations. The feature set may include a set of variables. A variable may be referred to as a feature. A specific observation may include a set of variable values corresponding to the set of variables. A set of variable values may be specific to an observation. In some cases, different observations may be associated with different sets of variable values, sometimes referred to as feature values.

In some implementations, the machine learning system may determine variables for a set of observations and/or variable values for a specific observation based on input received from the container management system. For example, the machine learning system may identify a feature set (e.g., one or more features and/or corresponding feature values) from structured data input to the machine learning system, such as by extracting data from a particular column of a table, extracting data from a particular field of a form and/or a message, and/or extracting data received in a structured data format. Additionally, or alternatively, the machine learning system may receive input from the container analyzeror from an operator to determine features and/or feature values.

In some implementations, the machine learning system may perform natural language processing and/or another feature identification technique to extract features (e.g., variables) and/or feature values (e.g., variable values) from text (e.g., unstructured data) input to the machine learning system, such as by identifying keywords and/or values associated with those keywords from the text.

As an example, a feature set for a set of observations may include a first feature of container dimensions, a second feature of container weight, a third feature of delivery day, and so on. As shown, for a first observation, the first feature may have a value of “3×3×4”, the second feature may have a value of “235”, the third feature may have a value of “Friday”, and so on. These features and feature values are provided as examples and may differ in other examples. For example, the feature set may include one or more of the following features: a weight of a container, a composition of a container, a material density of an item in the container, a composition of an item in the container, or a virtual 3D model of an item in the container, and a day of arrival of a container, a shipping carrier associated with the container, etc. In some implementations, the machine learning system may pre-process and/or perform dimensionality reduction to reduce the feature set and/or combine features of the feature set to a minimum feature set. A machine learning model may be trained on the minimum feature set, thereby conserving resources of the machine learning system (e.g., processing resources and/or memory resources) used to train the machine learning model.

The set of observations may be associated with a target variable. The target variablemay represent a variable having a numeric value (e.g., an integer value or a floating point value), may represent a variable having a numeric value that falls within a range of values or has some discrete possible values, may represent a variable that is selectable from one of multiple options (e.g., one of multiples classes, classifications, or labels), or may represent a variable having a Boolean value (e.g., 0 or 2, True or False, Yes or No, Male or Female), among other examples. A target variable may be associated with a target variable value, and a target variable value may be specific to an observation. In some cases, different observations may be associated with different target variable values.

In example, the target variableis an item identifier, which has a value of “Product #1” for both the first observation and the second observation. As discussed herein, a container handling location may receive multiples of the same item over time. For example, the container handling location may receive a recurring shipping of Product #1. Exampleemphasizes that the particular feature setmay vary across shipments, even when the same product is being delivery. This is due to many variables associated with shipping, such as available shipping supplies (e.g., box size), logistics delays, packaging density, etc. The feature set and target variable described above are provided as examples, and other examples may differ from what is described above. It will be understood that the target variable may vary across embodiments. For example, in some cases, the target variableis 3D model of an item in the container.

The target variable may represent a value that a machine learning model is being trained to predict, and the feature set may represent the variables that are input to a trained machine learning model to predict a value for the target variable. The set of observations may include target variable values so that the machine learning model can be trained to recognize patterns in the feature setthat lead to a target variable value. A machine learning model that is trained to predict a target variable value may be referred to as a supervised learning model or a predictive model. When the target variable is associated with continuous target variable values (e.g., a range of numbers), the machine learning model may employ a regression technique. When the target variable is associated with categorical target variable values (e.g., classes or labels), the machine learning model may employ a classification technique.

In some implementations, the machine learning model may be trained on a set of observations that do not include a target variable (or that include a target variable, but the machine learning model is not being executed to predict the target variable). This may be referred to as an unsupervised learning model, an automated data analysis model, or an automated signal extraction model. In this case, the machine learning model may learn patterns from the set of observations without labeling or supervision, and may provide output that indicates such patterns, such as by using clustering and/or association to identify related groups of items within the set of observations.

As further shown, the machine learning system may partition the set of observations into a training setthat includes a first subset of observations of the set of observations, and a test setthat includes a second subset of observations of the set of observations. The training setmay be used to train (e.g., fit or tune) the machine learning model, while the test setmay be used to evaluate a machine learning model that is trained using the training set. For example, for supervised learning, the test setmay be used for initial model training using the first subset of observations, and the test setmay be used to test whether the trained model accurately predicts target variables in the second subset of observations. In some implementations, the machine learning system may partition the set of observations into the training setand the test setby including a first portion or a first percentage of the set of observations in the training set(e.g., 75%, 80%, or 85%, among other examples) and including a second portion or a second percentage of the set of observations in the test set(e.g., 25%, 20%, or 25%, among other examples). In some implementations, the machine learning system may randomly select observations to be included in the training setand/or the test set.

As shown by reference number, the machine learning system may train a machine learning model using the training set. This training may include executing, by the machine learning system, a machine learning algorithm to determine a set of model parameters based on the training set. In some implementations, the machine learning algorithm may include a regression algorithm (e.g., linear regression or logistic regression), which may include a regularized regression algorithm (e.g., Lasso regression, Ridge regression, or Elastic-Net regression). Additionally, or alternatively, the machine learning algorithm may include a decision tree algorithm, which may include a tree ensemble algorithm (e.g., generated using bagging and/or boosting), a random forest algorithm, or a boosted trees algorithm. A model parameter may include an attribute of a machine learning model that is learned from data input into the model (e.g., the training set). For example, for a regression algorithm, a model parameter may include a regression coefficient (e.g., a weight). For a decision tree algorithm, a model parameter may include a decision tree split location, as an example.

As shown by reference number, the machine learning system may use one or more hyperparameter setsto tune the machine learning model. A hyperparameter may include a structural parameter that controls execution of a machine learning algorithm by the machine learning system, such as a constraint applied to the machine learning algorithm. Unlike a model parameter, a hyperparameter is not learned from data input into the model. An example hyperparameter for a regularized regression algorithm includes a strength (e.g., a weight) of a penalty applied to a regression coefficient to mitigate overfitting of the machine learning model to the training set. The penalty may be applied based on a size of a coefficient value (e.g., for Lasso regression, such as to penalize large coefficient values), may be applied based on a squared size of a coefficient value (e.g., for Ridge regression, such as to penalize large squared coefficient values), may be applied based on a ratio of the size and the squared size (e.g., for Elastic-Net regression), and/or may be applied by setting one or more feature values to zero (e.g., for automatic feature selection). Example hyperparameters for a decision tree algorithm include a tree ensemble technique to be applied (e.g., bagging, boosting, a random forest algorithm, and/or a boosted trees algorithm), a number of features to evaluate, a number of observations to use, a maximum depth of each decision tree (e.g., a number of branches permitted for the decision tree), or a number of decision trees to include in a random forest algorithm.

To train a machine learning model, the machine learning system may identify a set of machine learning algorithms to be trained (e.g., based on operator input that identifies the one or more machine learning algorithms and/or based on random selection of a set of machine learning algorithms), and may train the set of machine learning algorithms (e.g., independently for each machine learning algorithm in the set) using the training set. The machine learning system may tune each machine learning algorithm using one or more hyperparameter sets(e.g., based on operator input that identifies hyperparameter setsto be used and/or based on randomly generating hyperparameter values). The machine learning system may train a particular machine learning model using a specific machine learning algorithm and a corresponding hyperparameter set. In some implementations, the machine learning system may train multiple machine learning models to generate a set of model parameters for each machine learning model, where each machine learning model corresponds to a different combination of a machine learning algorithm and a hyperparameter setfor that machine learning algorithm.

In some implementations, the machine learning system may perform cross-validation when training a machine learning model. Cross validation can be used to obtain a reliable estimate of machine learning model performance using only the training set, and without using the test set, such as by splitting the training setinto a number of groups (e.g., based on operator input that identifies the number of groups and/or based on randomly selecting a number of groups) and using those groups to estimate model performance. For example, using k-fold cross-validation, observations in the training setmay be split into k groups (e.g., in order or at random). For a training procedure, one group may be marked as a hold-out group, and the remaining groups may be marked as training groups. For the training procedure, the machine learning system may train a machine learning model on the training groups and then test the machine learning model on the hold-out group to generate a cross-validation score. The machine learning system may repeat this training procedure using different hold-out groups and different test groups to generate a cross-validation score for each training procedure. In some implementations, the machine learning system may independently train the machine learning model k times, with each individual group being used as a hold-out group once and being used as a training group k−1 times. The machine learning system may combine the cross-validation scores for each training procedure to generate an overall cross-validation score for the machine learning model. The overall cross-validation score may include, for example, an average cross-validation score (e.g., across all training procedures), a standard deviation across cross-validation scores, or a standard error across cross-validation scores.

In some implementations, the machine learning system may perform cross-validation when training a machine learning model by splitting the training set into a number of groups (e.g., based on operator input that identifies the number of groups and/or based on randomly selecting a number of groups). The machine learning system may perform multiple training procedures and may generate a cross-validation score for each training procedure. The machine learning system may generate an overall cross-validation score for each hyperparameter setassociated with a particular machine learning algorithm. The machine learning system may compare the overall cross-validation scores for different hyperparameter setsassociated with the particular machine learning algorithm and may select the hyperparameter setwith the best (e.g., highest accuracy, lowest error, or closest to a desired threshold) overall cross-validation score for training the machine learning model. The machine learning system may then train the machine learning model using the selected hyperparameter set, without cross-validation (e.g., using all of data in the training setwithout any hold-out groups), to generate a single machine learning model for a particular machine learning algorithm. The machine learning system may then test this machine learning model using the test setto generate a performance score, such as a mean squared error (e.g., for regression), a mean absolute error (e.g., for regression), or an area under receiver operating characteristic curve (e.g., for classification). If the machine learning model performs adequately (e.g., with a performance score that satisfies a threshold), then the machine learning system may store that machine learning model as a trained machine learning modelto be used to analyze new observations, as described below in connection with.