Deployment to a Serving System for Online, Low-Latency Applications

A server may receive request messages from a user device over a network. The request messages indicate the data that is being requested by the user device. The server may generate data responsive to the request messages, and transmit response messages containing the data to the user device, over the network, to satisfy the requests of the user device. The user device may repeatedly transmit request messages if the response messages from the server contain insufficient, irrelevant, or otherwise unsatisfactory data with respect to the user device's requests. This cycle of request and response uses significant computing resources of the server and the user device in connection with message generation, as well as significant network resources in connection with message communication.

A serving system may manage requests from user devices. When a user device sends a request message over a network, the serving system processes this request and generates a response message. This response is then transmitted back to the user device to fulfill its request in real-time. For example, the user device's request may indicate particular data that is desired by the user device, and the serving system may generate the data and transmit a response that contains the data. However, if the response contains insufficient, irrelevant, or otherwise unsatisfactory data, the user device may repeatedly send additional request messages until its demands are fully met. This continuous cycle of request and response requires substantial computing power from both the serving system and the user device for generating and processing messages, and additionally requires significant network resources to handle the communication of these messages.

In some examples, the serving system may generate data for a response by inputting a request into a model that is trained to provide an output based on the request. To achieve low-latency processing of the request, while maintaining acceptable accuracy, the model may be trained using a technique known as knowledge distillation. In knowledge distillation, a large teacher model (e.g., in terms of number of parameters) can be trained using labeled training data. The teacher model may produce outputs having acceptable accuracy, but may be unsuitable for deployment into the serving system because of its large size and significant processing requirements, which could not satisfy the low-latency requirements of online, low-latency applications of the serving system. Thus, a smaller student model can be trained on the same labeled training data, but instead of learning directly from the training data, the student model learns to replicate the outputs of the teacher model.

Labeled training data may have a relatively limited supply compared to unlabeled training data, and therefore the student model, trained using limited labeled training data, may be insufficiently trained and unable to produce highly accurate and relevant outputs. Moreover, each data point in the training data may include data for multiple data fields (e.g., variables). In general, because the student model is trained using data for the multiple data fields, the serving system may also use the multiple data fields when inputting new data points into the student model. Without doing so, the student model may produce lower-quality outputs. However, the full range of data fields may include a significant amount of data, and inputting and processing this large amount of data may consume excessive processing resources and power of the serving system. This may produce additional latency, such that the requirements for online, low-latency applications of the serving system could not be met.

Systems and techniques described herein deploy a secondary (e.g., student) model, trained based on knowledge distillation from a primary (e.g., teacher) model using an unlabeled dataset, in a serving system to satisfy the low-latency requirements of the serving system. Using the secondary model, the serving system is able to reduce the processing resources, power resources, and network resources used by the serving system and user devices in connection with generating and exchanging network messages for online, low-latency applications of the serving system and achieving real-time responses to requests from the user devices.

The knowledge distillation provides a compression technique to compress the behavior of the larger primary model into the smaller secondary model. In some implementations, an offline system may train the primary model using labeled data, generate a large unlabeled dataset (e.g., using raw log data indicating historical user device requests), and train the secondary model based on knowledge distillation from the primary model using outputs from the primary model on the unlabeled dataset and a margin-based loss function (e.g., a margin mean squared error (MSE) loss function). Unlike conventional loss functions that rely on comparisons between data points labeled as relevant and data points labeled as irrelevant for a given request, the margin-based loss function can capture the relevance margin between any pair of data points for a given request, thereby eliminating the need for human labeled data. In this way, the secondary model can learn from the primary model using a much larger dataset. Expanding the dataset used to train the secondary model improves the performance of the secondary model. Accordingly, the secondary model can achieve a performance level that matches, or even surpasses, that of the primary model, while being lightweight enough for online, low-latency applications of serving systems. Moreover, the margin-based loss function enables the secondary model to learn effectively even if fewer data fields in the unlabeled dataset are used for the secondary model than are used for the primary model. This allows the secondary model to produce accurate outputs even if only a subset of data is inputted into the secondary model, thereby using less overall data and reducing consumption of processing resources.

In some implementations, the serving system may use the trained secondary model to generate high-quality data for responding to requests from user devices. For example, the serving system may receive a request from a user device in a network message, and the request may contain a request indicating particular data that is desired by the user device. Continuing with the example, the serving system may use the secondary model to generate high-quality data responsive to the request, and the serving system may transmit a response containing the data in a network message to the user device. By using the secondary model, which is relatively small and lightweight, the serving system may produce outputs with less latency and consuming less processing resources. Due to the way that the secondary model is trained, the serving system may obtain the high-quality data from the secondary model by inputting data reflecting only a subset of possible data fields (e.g., fewer data fields than are used to train the primary model). In this way, data processing in the serving system is reduced, thereby conserving significant processing resources and power. Moreover, due to the way that the secondary model is trained, the serving system may provide more relevant and accurate data to the user device that satisfies the user device's request. By doing so, fewer requests and responses are exchanged in network messages between the serving system and the user device. In this way, the serving system conserves processing resources and network resources that otherwise would be used in connection with processing additional requests, generating additional network messages relating to the additional requests, and transmitting the additional network messages.

Some implementations described herein relate to a system for deploying a secondary model in a serving system that is configured to manage request traffic made in a network by user devices. The system may include one or more memories and one or more processors communicatively coupled to the one or more memories. The one or more processors may be configured to cause the system to receive, from one or more servers, raw log data indicating a historical request and a candidate result set corresponding to the historical request. The one or more processors may be configured to cause the system to generate, based on the historical request and the candidate result set, a first version of an unlabeled dataset that indicates a plurality of request-result pairs using a set of data fields used to train a primary model, and a second version of the unlabeled dataset that indicates the plurality of request-result pairs using a subset of the set of data fields that is smaller in size than the set of data fields. The one or more processors may be configured to cause the system to execute a process for adjusting parameters of a secondary model using the first version of the unlabeled dataset as input into the primary model to provide labels for the plurality of request-result pairs, using the second version of the unlabeled dataset as input into the secondary model, and using a margin-based loss function on all candidate result pairs of the candidate result set for the historical request. The one or more processors may be configured to cause the system to deploy, to the serving system, the secondary model for handling network requests from user devices.

Some implementations described herein relate to a method of managing request traffic in a serving system. The method may include receiving, by a device of the serving system and from a user device, a request. The method may include determining, by the device and based on the request, a retrieval set that includes a plurality of candidate results for the request. The method may include generating, by the device, a plurality of request-result pairs based on the request and the retrieval set. The method may include determining, by the device, relevance outputs for the plurality of request-result pairs using a secondary model that is trained based on a primary model using: (1) an unlabeled dataset that is based on a candidate result set for a historical request, and (2) a margin-based loss function on all candidate result pairs of the candidate result set for the historical request. The method may include transmitting, by the device to the user device and in accordance with the relevance outputs, data for presentation in a user interface of the user device based on the request.

Some implementations described herein relate to an apparatus. The apparatus may include means for generating an unlabeled dataset that includes a plurality of request-result pairs based on a historical request and a candidate result set associated with the historical request. The apparatus may include means for training a secondary model, based on a primary model, using the unlabeled dataset as input to the primary model and to the secondary model, and using a loss function on all candidate result pairs in the candidate result set for the historical request. The apparatus may include means for deploying the secondary model for handling requests from user devices.

The following detailed description of example implementations refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements.

1 1 FIGS.A-F 100 100 105 110 115 120 125 illustrate an example environmentfor deployment of a secondary model for online, low-latency applications. As shown, the environmentincludes an offline system, a serving system, a data node, a user device, and a network.

100 125 125 125 105 110 105 110 125 Communication among the devices and systems of environmentmay be performed via the network. The networkmay include one or more wired and/or wireless networks. For example, the networkmay include a wireless wide area network (e.g., a cellular network or a public land mobile network), a local area network (e.g., a wired local area network or a wireless local area network (WLAN), such as a Wi-Fi network), a personal area network (e.g., a Bluetooth network), a near-field communication network, a telephone network, a private network, the Internet, and/or a combination of these or other types of networks. In some implementations, where the offline systemand the serving systemare components of the same higher-level system, the offline systemand the serving systemmay communicate without use of the network.

115 115 115 115 105 110 125 115 105 110 125 The data nodemay include storage, one or more data sources, one or more data services (e.g., accessible via an application programming interface (API)), and/or one or more databases, among other examples. The data nodemay include, or may be included in, one or more servers. The data nodemay store information relating to labeled training data, raw data logs for a search system, and/or frequently-used request-result pairs, among other examples. The data nodemay provide any of the aforementioned information to the offline systemand/or to the serving system(e.g., via the network). In some implementations, communications of the data nodewith the offline systemand/or the serving systemmay be performed locally (e.g., within a cloud computing environment) rather than through the network.

120 120 120 120 110 110 120 120 The user devicemay include a communication device and/or a computing device. For example, the user devicemay include a wireless communication device, a mobile phone, a user equipment, a laptop computer, a tablet computer, a desktop computer, a gaming console, a set-top box, a wearable communication device (e.g., a smart wristwatch, a pair of smart eyeglasses, a head mounted display, or a virtual reality headset), or a similar type of device. The user devicemay implement a user interface facilitating user-computer interaction and communication, and may include display screens, keyboards, a mouse, and the appearance of a desktop. For example, a user interface may include a way a user interacts with an application or a website. The user devicemay exchange data with the serving system. For example, data exchanged with the serving systemmay be in connection with the user deviceaccessing a web page using a web browser installed on the user device.

105 105 The offline systemmay include a communication device and/or a computing device. For example, the offline systemmay include one or more physical servers, one or more virtual servers (e.g., executing on computing hardware), and/or cloud computing resources (e.g., executing on computing hardware used in a cloud computing environment). “Offline” may refer to an environment in which the execution of data processing tasks is performed independently from real-time user inputs.

110 110 110 110 120 120 125 110 110 120 The serving systemmay include a communication device and/or a computing device. For example, the serving systemmay include one or more physical servers, one or more virtual servers (e.g., executing on computing hardware), and/or cloud computing resources (e.g., executing on computing hardware used in a cloud computing environment). In some implementations, the serving systemmay be implemented on an edge node of a network or of a cloud system (e.g., in a regional data center of the cloud system). In some implementations, the serving systemmay be a component (e.g., a hardware component or a software component) of the user device. The serving system may be configured to manage request traffic from user devicesmade via the network. The serving systemmay be an online system. “Online” may refer to an environment in which the execution of data processing tasks are performed in response to real-time user inputs. For example, the serving systemmay provide responses to requests from user devicesin real time. “Real time” may refer to the responses being provided within a time frame (e.g., within 2 seconds, within 1 second, or within 100 milliseconds) that is perceptibly concurrent or near-concurrent with the requests.

105 110 125 105 110 105 110 120 105 110 In some implementations, communications between the offline systemand the serving systemmay be performed through the network. For example, the offline systemmay be a centralized processing system located on dedicated server(s) or cloud computing environment(s) that have a significant amount of computing resources, to perform the actions described below, that are not available or at least cannot be routinely utilized by the serving systemin the same way. The offline systemmay communicate with and perform the same role for multiple serving systemsthat may be distributed in different geographic areas and/or communicate with different groups of user devices. In this way, the offline systemmay reduce the processing resources that are utilized compared to, for example, each serving systemhaving to have the same actions performed by a dedicated offline system.

105 110 125 105 110 105 110 105 110 In other implementations, communications between the offline systemand the serving systemmay be performed locally (e.g., within a cloud computing environment) rather than through the network. For example, the offline systemand the serving systemmay be implemented in the same cloud computing resource or in separate cloud computing resources within the same cloud computing environment. In some implementations, the offline systemand the serving systemmay be components (e.g., co-located components or distributed components) of the same higher-level processing system. For example, the offline systemand the serving systemmay be hardware components or software components of the same system.

105 130 135 130 135 135 130 The offline systemmay perform training of a primary model(also referred to as a “teacher model”) and a secondary model(also referred to as a “student model”). The primary modeland the secondary modelmay be machine learning models. The secondary modelmay be based on a large language model (LLM) that is significantly smaller than an LLM of the primary model(e.g., in terms of parameters), such as 60 times smaller or more. A “large language model” or “LLM” may refer to a computational system designed to understand and generate human language through learning patterns in vast amounts of training data, such as books, websites, and other sources.

105 135 130 135 130 130 135 130 130 135 135 135 130 135 The offline systemmay train the secondary modelthrough capability transfer from the primary model. For example, the training of the secondary modelmay be based on knowledge distillation from the primary model. This knowledge distillation may involve training the primary model, and training the secondary modelto mimic the results of the primary model. For example, the knowledge distillation described herein may enable distilling an LLM of the primary modelinto an LLM of the secondary model, such that the secondary modelprovides reliable outputs while maintaining high efficiency (e.g., the secondary modelmay include an LLM that is at least as effective as the LLM of the primary model). In this way, the smaller secondary modelresolves latency issues that generally prevent LLMs from being utilized in real-time systems.

135 110 135 120 130 135 Thus, the secondary modelcan be deployed in the serving systemto handle requests (e.g., network requests) from user devices and return relevant results (e.g., the secondary modelmay be a ranking model for search queries). A request may include information indicating data (e.g., a type of data) that the user devicewants. For example, a request may include a query (e.g., a search query), which may be information (e.g., text, such as a word or a phrase) used to retrieve information from an information retrieval system (e.g., a search system). For example, a query may be text entered into a search input of a web page. A “candidate result” may refer to a result that has the potential to be responsive to (e.g., relevant to) a request. For example, the candidate result may be an electronic document (e.g., a web page) or an item (e.g., that is described in an electronic document). Thus, for a given request q and candidate result d, their predicted relevance by the primary modelmay be denoted t(q, d)∈and their predicted relevance by the secondary modelmay be denoted s(q, d)∈.

1 FIG.A 150 105 115 125 135 As shown in, and by reference number, the offline systemmay receive a labeled dataset from the data node(e.g., via the network). A “labeled dataset” refers to a dataset that includes both input data and corresponding correct output labels used in supervised learning for a machine learning model. The labeled dataset may include a plurality of request-result pairs and soft relevance labels (also referred to herein as “soft targets”) for the request-result pairs. A “soft relevance label” may refer to a probabilistic value that indicates relative relevance (e.g., rather than a binary classification) in order to provide guidance to the secondary model. Each request-result pair may include a combination (e.g., a concatenation) of a request (e.g., the textual content of the request) and a set of data fields for a single candidate result (e.g., a request-result pair is a text string). The set of data fields may indicate metadata relating to a candidate result (e.g., relating to an item described in an electronic document). For example, the set of data fields may include an item title field, an item type field, an item brand field, an item color field, an item gender field, and/or an item description field.

For the labeled dataset, each request-result pair (q, d) may be given a rating. For example, ratings may be 4 (e.g., excellent; perfect match), 3 (e.g., good; item with a mismatched attribute), 2 (e.g., okay), 1 (e.g., bad; irrelevant), or 0 (e.g., embarrassing). The ratings may be mapped to classes (e.g., Class 2, Class 1, and Class 0), thereby providing a class label for each request-result pair (q, d). For example, rating 4 may be mapped to Class 2, rating 3 may be mapped to Class 1, and ratings 2, 1, and 0 may be mapped to Class 0. The relevance prediction may be formulated as a binary classification problem with soft targets. Thus, the classes may be mapped to soft targets. For example, Class 2 may be mapped to a soft target value of 1, Class 1 may be mapped to a soft target value of 0.5, and Class 0 may be mapped to a soft target value of 0. In this way, the soft targets can be converted from the editorial rating labels.

152 105 130 105 130 105 130 130 130 130 130 130 teacher As shown by reference number, the offline systemmay execute a process for adjusting parameters of the primary model. For example, the offline systemmay train the primary modelto determine relevance (e.g., a level of relevance) between requests and candidate results. The offline systemmay train the primary modelusing the labeled dataset. Request-result pairs in the labeled dataset that are inputted into the primary modelmay be tokenized. The primary modelmay include an LLM that is fine-tuned using a cross-encoder architecture to predict the relevance between a request and a candidate result. The primary modelmay be trained as a classification model using request-result pairs as inputs along with soft relevance labels. Classification may be based on the “last token,” where the hidden state of this last token is used as the representation of a request-result pair. The primary model may include a multilayer perceptron (MLP) layer to transform this representation into a single output value, which represents the predicted relevance score, t(q, d), of the primary model. The primary modelmay be trained using cross-entropy loss, as defined by(q) for each request q:

q q,d q,d where Dis the total candidate result collection for request q, and yis the soft label for request q and candidate result d (e.g., y∈{0, 0.5, 1}). The objective of the training may be to optimize t(q, d) by minimizing the loss function.

154 105 135 135 135 105 110 115 105 105 110 105 In some implementations, as shown by reference number, the offline systemmay generate an unlabeled dataset for training the secondary model. For example, the labeled dataset may provide insufficient training data for the secondary model. Moreover, the labeled dataset may lack sufficient negative-label examples and/or may include incorrect labels, thereby affecting its usability for training the secondary model. The offline systemmay receive raw log data (e.g., request log data associated with the serving system) from the data node, and the offline systemmay generate the unlabeled dataset using raw log data. For example, the offline systemmay generate the unlabeled dataset using a plurality of historical requests (e.g., made at the serving system) and a plurality of retrieval sets of candidate results associated with the historical requests. For example, each historical request may be associated with a set of relevant candidate results, retrieved from a total set of candidate results (e.g., an entire item catalog), that form a retrieval set for that request. The offline systemmay generate a plurality of request-result pairs based on the historical requests and the retrieval sets. Accordingly, the unlabeled dataset may include this plurality of request-result pairs without labels (e.g., without soft relevance labels, as in the case of the labeled dataset).

156 105 135 105 135 135 135 135 135 130 135 130 As shown by reference number, the offline systemmay execute a process for adjusting parameters of the secondary model. For example, the offline systemmay train the secondary modelusing the unlabeled dataset. Using the much larger unlabeled dataset improves the training of the secondary modelcompared to training the secondary modelbased on the smaller labeled dataset, thereby improving the accuracy and reliability of the secondary model. The training of the secondary modelmay be based on knowledge distillation from the primary model. Knowledge distillation may be used to train the secondary modelto mimic the outputs of the primary model.

135 130 130 135 135 130 135 The knowledge distillation may use a margin-based loss function (e.g., a margin MSE loss function) on all candidate result pairs (i.e., a pair of candidate results) for a given request (e.g., not merely relevant-irrelevant candidate result pairs for the request). Using margin MSE to align the margins predicted by the secondary modelwith that of the primary modelsignificantly improves knowledge distillation compared to using pointwise cross-entropy loss (e.g., even when both the primary modeland the secondary modelhave cross-encoder architectures). As an example, the secondary modelmay be trained to mimic the margin between a pair of candidate results that is predicted by the primary model. Rather than distilling knowledge by directly optimizing raw scores for each request-result pair, margin MSE loss uses the margin between pairs of candidate results for each request, thereby facilitating effective training using unlabeled data (e.g., where all request-result pairs do not have known true labels, such as “relevant” or “irrelevant”). In this way, the secondary model, trained using the margin-based loss function, is capable of providing more relevant and accurate outputs.

105 135 130 135 135 130 130 135 1 FIG.B 1 FIG.B The offline systemmay train the secondary modelusing the unlabeled dataset as input into the primary modeland into the secondary model, and using the margin-based loss function (e.g., the margin MSE loss function), as shown in. In particular,illustrates a framework for training the secondary modelusing knowledge distillation from the primary model. In some implementations, the primary modeland the secondary modelmay have cross-encoder architectures.

135 130 160 162 164 162 164 160 162 164 160 160 130 135 160 162 160 162 130 135 160 164 160 164 i j Training the secondary modelusing the primary modelmay involve inputs of a request(q), a first candidate result(d), and a second candidate result(d). The first candidate resultand the second candidate resultmay include any pair of candidate results from a collection of candidate results, such as a retrieval set for the request. For example, the first candidate resultand the second candidate resultdo not need to include one relevant candidate result to the requestand one irrelevant candidate result to the request. As shown, the primary modeland the secondary modelmay be applied to the requestand the first candidate result(e.g., a request-result pair that is a combination of the requestand the set of data fields for the first candidate result, as described herein), and a mirrored primary modeland secondary modelmay be applied to the requestand the second candidate result(e.g., a request-result pair that is a combination of the requestand the set of data fields for the second candidate result, as described herein).

130 131 131 131 131 131 131 132 130 132 The primary modelmay include a language model component(e.g., an LLM) as encoder. For example, the language model componentmay be a bidirectional-encoder-representations-from-transformers model. In some implementations, the language model componentmay utilize one billion or more parameters. For example, the language model componentmay utilize approximately seven billion parameters. The output of the language model componentmay be a multi-dimensional vector, which represents a request-result pair. The multi-dimensional vector output of the language model componentmay be input to an MLP componentof the primary model. The MLP componentmay output a single number representing a prediction of a level of relevance for the request-result pair (e.g., a relevance of the candidate result to the request). This single number prediction can be converted to a 0 to 1 scale (e.g., using a sigmoid function). The single output may enhance the effectiveness of the knowledge distillation.

135 136 136 136 135 131 130 136 136 135 136 137 137 138 139 The secondary modelmay include a language model component(e.g., an LLM) as encoder. For example, the language model componentmay be a bidirectional-encoder-representations-from-transformers model. The language model componentof the secondary modelmay use fewer parameters than the language model componentof the primary model. In some implementations, the language model componentmay utilize 150 million or fewer parameters. For example, the language model componentmay utilize approximately 110 million parameters. As a result of the techniques described herein, additional model capacity may not be necessary for successful knowledge distillation. A request-result pair may be input to the secondary modelusing a CLS token, which may refer to a classification token added at the beginning of the request-result pair sequence. The language model componentmay output a representation, such as a multi-dimensional vector or embedding, of the input tokens. After the encoder pass, the hidden state of the CLS token is used as a representation of the request-result pair. An MLP componentis then applied to this hidden state. In some implementations, an output of the MLP componentmay be provided to a dropout layerand then a classification headbefore arriving at a final relevance output.

135 166 130 130 135 135 166 i j i j i j The secondary modelmay be trained using a loss functionbased on an output t(q, d) of the primary model, an output t(q, d) of the mirrored primary model, an output s(q, d) of the secondary model, and an output s(q, d) of the mirrored secondary model. The loss functionmay compute a loss l(q, d, d) as:

135 135 i j i j q q q q q + − + − As an example, training the secondary modelusing a margin MSE loss may include computing the margin MSE loss for all triplets (q, d, d), wherein dand dare candidate results from a collection (e.g., a retrieval set) for request q, denoted by D. |D| may denote the number of candidate results in D, and thereby the number of triplets may be represented as [D](|D|−1)/2. In contrast to other approaches that use triplets (q, d, d), where dand dare a relevant document and an irrelevant document, respectively, in accordance with relevance labels for training data, margin MSE enables knowledge distillation from large unlabeled datasets (e.g., that lack editorial labels). Using margin MSE, the goal during training of the secondary modelis to minimize a difference between (e.g., make as close as possible)

where

i i 135 130 represent relevance score differences for the request-result pairs (q, d) and (q, d) from the secondary modeland the primary model, respectively. In particular,

135 The margin MSE loss function used by the secondary modelfor request q may be defined as:

The margin MSE loss may be computed on all candidate result pairs from a candidate result collection for a given request, thereby eliminating the need for true labels.

135 105 130 130 130 135 135 135 130 135 130 i j For example, to train the secondary model, the offline systemmay sample (e.g., randomly or without restriction), for a given request, any pair of candidate results (d, d) from the collection of candidate results and use the corresponding request-result pairs in the unlabeled dataset as input into the primary model(e.g., into the primary modeland a mirror of the primary model) and into the secondary model(e.g., into the secondary modeland a mirror of the secondary model). The primary modeland the secondary modelmay output respective relevance outputs (e.g., relevance scores) for the request-result pairs of the unlabeled dataset. The relevance outputs from the primary modeleffectively provide labels for request-result pairs that can be used in loss calculation. A relevance output of a request-result pair may indicate a level of relevance between a request and a candidate result that make up the request-result pair. In some implementations, a relevance output may be a value between 0 and 1, inclusive.

105 130 135 130 105 105 135 The offline systemmay apply the margin MSE loss function to the relevance outputs from the primary modeland the relevance outputs from the secondary modelto determine a loss value. For example, the margin MSE loss function may be based on a difference between first relevance outputs of the primary modelfor a first request-result pair and a second request-result pair for the given request, and a difference between second relevance outputs of the secondary model for the first request-result pair and the second request-result pair for the given request. Because the pair of candidate results may be sampled from any two documents in the collection, the first request-result pair and the second request-result pair may relate to a pair of relevant candidate results to the given request (e.g., both candidate results are relevant to the request), or may relate to a pair of irrelevant candidate results to the given request (e.g., both candidate results are irrelevant to the request). In other words, it is not required for the sampling to select one relevant candidate result and one irrelevant candidate result, although this may occur by chance in some cases. The offline systemmay perform this process iteratively for all pairs of candidate results (e.g., every pairwise combination) from the collection of candidate results for the given request in order to determine an overall loss value. The offline systemmay adjust parameters of the secondary modelbased on the overall loss value or the individual loss values for each pair of candidate results (e.g., using backpropagation), with the goal of minimizing the loss value.

130 135 130 135 105 130 135 130 135 130 135 105 135 130 135 130 135 In some implementations, the unlabeled dataset input into the primary modelmay include request-result pairs based on the set of data fields, and the unlabeled dataset input into the secondary modelmay include request-result pairs based on a smaller subset of the set of data fields (i.e., a subset smaller in size than the set of data fields). For example, the request-result pairs in the unlabeled dataset used for the primary modelmay include a long text field (e.g., of 150 characters or more), such as an item description field, whereas the request-result pairs in the unlabeled dataset used for the secondary modelmay not include the long text field. In some implementations, the offline systemmay generate a first version of the unlabeled dataset using the entire set of data fields for the primary model, and generate a second version of the unlabeled dataset using the smaller subset of data fields for the secondary model. The first version and the second version of the unlabeled dataset may be generated prior to being input into the primary modeland the secondary model, respectively, or may be generated in real time as they are input into the primary modeland the secondary model. In some other examples, the offline systemmay train the secondary modelusing the same version of the unlabeled dataset as inputs into the primary modeland the secondary model(e.g., the number of input features may be the same for the primary modeland the secondary model).

105 130 135 135 135 105 130 130 135 135 In some examples, the offline systemmay input the unlabeled dataset into the primary modeland into the secondary modelconcurrently during training of the secondary model. In some other examples, prior to training the secondary model, the offline systemmay input the unlabeled dataset into the primary modelto obtain relevance outputs, and then label the request-result pairs of the unlabeled dataset using the relevance outputs to obtain an artificially-labeled dataset (e.g., a dataset labeled by the primary modelrather than a human). The artificially-labeled dataset can be used for comparison to relevance outputs of the secondary modeland computation of a loss value during training of the secondary model.

1 FIG.C 170 105 135 110 110 135 135 135 135 135 110 135 135 110 110 135 135 135 135 As shown in, and by reference number, the offline system, or its higher-level system, may deploy the secondary modelin the serving system, thereby enabling the serving systemto handle requests (e.g., real-time requests) using the secondary model. Deploying the secondary modelmay include converting the secondary modelinto a deployment format, such as by transforming the secondary model's parameters and architecture into a binary format. Deploying the secondary modelmay also include loading (e.g., installing) dependencies and libraries for the secondary modelin the serving system. Deploying the secondary modelmay further include loading (e.g., installing) the secondary model(e.g., in the deployment format) in the serving system. In some implementations, the serving systemmay implement an API used to handle incoming requests and pass them to the secondary modelfor inference. The API may be existing at a time of the deployment of the secondary model, or deploying the secondary modelmay include configuring and loading API components for the secondary model.

135 110 110 135 172 110 120 120 Once the trained secondary modelis deployed to the serving system, the serving systemmay use the trained secondary modelto handle incoming requests. As shown by reference number, the serving systemmay receive, from the user device, a request (e.g., indicating a search query). For example, the request from the user devicemay be a network request in a network message, such as an HTTP request. In some implementations, the request may indicate a query, such as a “tail query.” Search queries are often segmented into head, torso, and tail queries. Head and torso queries generally provide enough engagement data to train machine learning models for retrieving and reranking relevant items. However, it is difficult to effectively retrieve and rerank the most relevant documents for tail queries due to the lack of engagement data.

120 110 120 120 120 120 120 120 120 120 120 110 120 120 110 110 In connection with the request from the user device, the serving systemmay receive information regarding the user device(e.g., the request may include the information). The information regarding the user devicemay indicate a technical platform of the user device, such as information indicating a hardware model number or generation of the user device, a display size of the user device, a display resolution of the user device, a processor type of the user device, an operating system of the user device, and/or an application used by the user deviceto transmit the request (e.g., a dedicated mobile application that is for a particular type of mobile operating system and is configured to specifically interact with the serving system, a mobile web browser, a desktop web browser, etc.), among other examples. In some examples, the information regarding the user devicemay indicate interactions of the user device(e.g., interactions with user interface elements of the serving system, resources accessed in the serving system, or the like).

1 FIG.D 174 110 110 As shown in, and by reference number, in a retrieval stage of processing the request, the serving systemmay determine, based on the request, a retrieval set (e.g., a recall set) that includes a plurality of candidate results. The retrieval set may be a smaller subset from a collection of candidate results. For example, the collection of candidate results may include all candidate results (e.g., all items in a catalog), and the retrieval set may include less than all the candidate results from the collection. The candidate results in the retrieval set may be ranked, but this may be a preliminary ranking for the candidate results. In some implementations, the serving systemmay determine the retrieval set (e.g., the retrieval set and its preliminary ranking) based on keyword matching, or another simple, low-latency technique for determining initial candidate results for the request.

176 110 135 135 130 110 135 As shown by reference number, the serving systemmay generate a plurality of request-result pairs based on the request and the retrieval set, which can be used as an input for the secondary model. For example, each request-result pair may include a combination (e.g., a concatenation) of the request and a smaller subset of data fields for a respective candidate result of the retrieval set (e.g., the combination thus enabling the secondary modelto benefit from the attention mechanism across all tokens in the input). The smaller subset of data fields may be a smaller subset of the set of data fields used for training the primary model(e.g., lacking a long text field, such as an item description field), as described herein. By using a smaller subset of data fields, the serving systemprocesses much less data using when using the secondary model, thereby conserving significant processing resources and power.

135 136 135 135 t p b c g t p b c d [CLS] query [SEP] title [SEP] type [SEP] brand [SEP] color [SEP] genderwhere [CLS] is a classification token added at the beginning of the request-result pair sequence and representative of the entire sequence (e.g., representative of the request-result pair), [SEP], [SEP], [SEP], [SEP], and [SEP] represent distinct and unused separator tokens, and title, type, brand, color, and gender represent data fields. In some implementations, a request-result pair for inputting to the secondary model(e.g., at inference or during training) may include the request (e.g., textual content of the request) concatenated with the smaller subset of data fields, where the request is separated from the data fields by a separator token and the data fields are separated from each other by separator tokens. Each separator token may be a distinct and unused token in a vocabulary of the language model componentof the secondary model. For example, a request-result pair for inputting to the secondary modelmay be in the form:

1 FIG.E 178 110 135 110 As shown in, and by reference number, in a ranking stage of processing the request (also referred to as a “re-ranking stage”), the serving systemmay input the generated request-result pairs (e.g., embeddings representing the request-result pairs) into the secondary modelto obtain relevance outputs for the request-result pairs. For example, a relevance output (e.g., a relevance score) for a request-result pair may indicate a level of relevance of the candidate result, of the request-result pair, to the request. Accordingly, the serving systemmay generate a relevance ranking (e.g., a re-ranking, relative to the initial ranking of the retrieval set) of the request-result pairs (e.g., a ranking of the request-result pairs from most relevant to least relevant) based on the relevance outputs.

1 FIG.F 180 110 120 110 110 120 As shown in, by reference number, the serving systemmay transmit, in accordance with the relevance outputs (e.g., in accordance with the relevance ranking), a response indicating data for presentation in a user interface of the user device. The response may be a network response in a network message, such as an HTTP response. The data may indicate one or more candidate results (e.g., at least a most-relevant candidate result). In some implementations, the data may be formatted as search results for presentation in a web page. In some implementations, the relevance outputs may be one of multiple features used by the serving systemto determine a final relevance ranking of the candidate results. For example, the serving systemmay determine the final relevance ranking based on the relevance outputs and the information regarding the user device.

120 120 110 110 120 120 As an example, the information regarding the user devicemay indicate interactions of the user device(e.g., clicks) with user interface elements (e.g., representing candidate results) in the serving system. Accordingly, the serving systemmay determine the final relevance ranking based on the relevance outputs and the relevance of candidate results to the interactions of the user device(e.g., relevance of the candidate results to previous candidate results that the user devicehas had interactions with).

120 120 120 110 110 Additionally, or alternatively, the information regarding the user devicemay include, for example, whether the user deviceused a first type of mobile application for a first type of mobile operating system, a second type of mobile application for a second type of mobile operating system, a first type of mobile browser for the first type of mobile operating system, a second type of mobile browser for a second type of mobile operating system to generate the request, and/or a particular type of browser for a desktop computer. For example, different candidate results may be relevant to, or suitable for, different technical platforms and/or different interaction histories. As an example, if the user deviceis using the first type of mobile application, then the serving systemmay determine the final relevance ranking based on relevance to the first type of mobile application. Continuing with the example, the serving systemmay determine a different final relevance ranking for the second type of mobile application.

135 110 105 110 135 110 105 105 105 135 105 115 In some implementations, based on a use of the secondary modelin the serving system, the offline system, the serving system, or their higher-level system, may collect data relating to usage counts for request-result pairs. For example, a usage count may indicate the number of times that a particular request-result pair has been input into the secondary modelin connection with request processing at the serving system. The offline systemmay then identify one or more frequently-used request-result pairs associated with a request (e.g., based on the usage counts for request-result pairs). For example, the offline systemmay identify a frequently-used request-result pair based on its usage count satisfying a threshold. Based on identifying the frequently-used request-result pair(s), the offline systemmay input the frequently-used request-result pair(s) into the secondary modelto obtain relevance outputs for the frequently-used request-result pair(s). The offline systemmay store these relevance outputs, in association with the frequently-used request-result pair(s), in a cache (e.g., of data node).

110 172 110 110 110 135 110 110 135 135 110 135 In some implementations, the request received by the serving systemat reference numbermay be the same as the request associated with the frequently-used request-result pair(s). Here, in response to receiving the request, the serving systemmay retrieve the relevance outputs for the frequently-used request-result pair(s) from the cache. The cached relevance outputs minimize the need for real-time computation, thereby reducing repetitive processing in the serving systemand conserving processing resources. In some examples, each candidate result in a retrieval set for the request may have a corresponding frequently-used request-result pair and relevance output retrieved from the cache. Accordingly, the serving systemmay determine a relevance ranking for request-result pairs from the retrieval set for the request using the relevance outputs retrieved from the cache, and without using the secondary model. In some other examples, the serving systemmay identify one or more request-result pairs from the retrieval set for the request that do not have corresponding frequently-used request-result pairs and relevance outputs in the cache. Accordingly, the serving systemmay invoke the secondary modeland input these request-result pairs into the secondary modelto obtain supplemental relevance outputs for the request-result pairs. The serving systemmay then determine a relevance ranking for request-result pairs from the retrieval set for the request using relevance outputs retrieved from the cache (for those request-result pairs with corresponding relevance outputs in the cache) and the supplemental relevance outputs obtained from the secondary model(for those request-result pairs without corresponding relevance outputs in the cache).

135 130 135 135 130 110 130 135 110 135 130 135 135 110 The techniques described herein facilitate training of the secondary modelusing an unlabeled dataset, which can be much larger than the labeled dataset used to train the primary model. The large unlabeled dataset, when used with the margin-based loss function that accounts for the lack of true labels in the unlabeled dataset, improves the training of the secondary model. Accordingly, the secondary modelmay be capable of producing relevant and accurate outputs, that match or exceed those of the primary model, while being able to satisfy the low-latency requirements of online, low-latency applications of the serving systemdue to being significantly smaller than the primary model. The relevant and accurate outputs produced by the secondary modelenable the serving systemto respond to requests from user devices with accurate, high-quality data that satisfies the user devices' requests. By doing so, fewer requests and responses are exchanged in network messages between the serving system and user devices, thereby conserving significant processing resources and network resources. Furthermore, the secondary modelis capable of producing relevant and accurate outputs even when fewer data fields (e.g., relative to the amount of data fields used in training of the primary model) are provided to the secondary modelas input. Thus, through reducing the amount of input data that is needed by the secondary modelin order to produce relevant and accurate outputs, the serving systemis able to conserve significant processing resources and power.

1 1 FIGS.A-F 1 1 FIGS.A-F As indicated above,is provided as an example. Other examples may differ from what is described with regard to.

2 FIG. 2 FIG. 200 200 105 110 115 120 105 110 115 120 200 200 200 210 220 230 240 250 260 210 220 230 240 250 260 is a diagram of example components of a deviceassociated with deployment of a secondary model for online, low-latency applications. The devicemay correspond to the offline system, serving system, data node, and/or user device. In some implementations, offline system, serving system, data node, and/or user devicemay include one or more devicesand/or one or more components of the device. As shown in, the devicemay include a bus, a processor, a memory, an input component, an output component, and/or a communication component. The bus, the processor, the memory, the input component, the output component, and/or the communication componentmay provide means for performing one or more operations described herein.

210 200 210 210 220 220 220 2 FIG. The busmay include one or more components that enable wired and/or wireless communication among the components of the device. The busmay couple together two or more components of, such as via operative coupling, communicative coupling, electronic coupling, and/or electric coupling. For example, the busmay include an electrical connection (e.g., a wire, a trace, and/or a lead) and/or a wireless bus. The processormay include a central processing unit, a graphics processing unit, a microprocessor, a controller, a microcontroller, a digital signal processor, a field-programmable gate array, an application-specific integrated circuit, and/or another type of processing component. The processormay be implemented in hardware, firmware, or a combination of hardware and software. In some implementations, the processormay include one or more processors capable of being programmed to perform one or more operations or processes described elsewhere herein.

230 230 230 230 230 200 230 220 210 220 230 220 230 230 The memorymay include volatile and/or nonvolatile memory. For example, the memorymay include random access memory (RAM), read only memory (ROM), a hard disk drive, and/or another type of memory (e.g., a flash memory, a magnetic memory, and/or an optical memory). The memorymay include internal memory (e.g., RAM, ROM, or a hard disk drive) and/or removable memory (e.g., removable via a universal serial bus connection). The memorymay be a non-transitory computer-readable medium. The memorymay store information, one or more instructions, and/or software (e.g., one or more software applications) related to the operation of the device. In some implementations, the memorymay include one or more memories that are coupled (e.g., communicatively coupled) to one or more processors (e.g., processor), such as via the bus. Communicative coupling between a processorand a memorymay enable the processorto read and/or process information stored in the memoryand/or to store information in the memory.

240 200 240 250 200 260 200 260 The input componentmay enable the deviceto receive input, such as user input and/or sensed input. For example, the input componentmay include a touch screen, a keyboard, a keypad, a mouse, a button, a microphone, a switch, a sensor, a global positioning system sensor, a global navigation satellite system sensor, an accelerometer, a gyroscope, and/or an actuator. The output componentmay enable the deviceto provide output, such as via a display, a speaker, and/or a light-emitting diode. The communication componentmay enable the deviceto communicate with other devices via a wired connection and/or a wireless connection. For example, the communication componentmay include a receiver, a transmitter, a transceiver, a modem, a network interface card, and/or an antenna.

200 230 220 220 220 220 200 220 The devicemay perform one or more operations or processes described herein. For example, a non-transitory computer-readable medium (e.g., memory) may store a set of instructions (e.g., one or more instructions or code) for execution by the processor. The processormay execute the set of instructions to perform one or more operations or processes described herein. In some implementations, execution of the set of instructions, by one or more processors, causes the one or more processorsand/or the deviceto perform one or more operations or processes described herein. In some implementations, hardwired circuitry may be used instead of or in combination with the instructions to perform one or more operations or processes described herein. Additionally, or alternatively, the processormay be configured to perform one or more operations or processes described herein. Thus, implementations described herein are not limited to any specific combination of hardware circuitry and software.

2 FIG. 2 FIG. 200 200 200 The number and arrangement of components shown inare provided as an example. The devicemay include additional components, fewer components, different components, or differently arranged components than those shown in. Additionally, or alternatively, a set of components (e.g., one or more components) of the devicemay perform one or more functions described as being performed by another set of components of the device.

3 FIG. 3 FIG. 3 FIG. 3 FIG. 300 105 110 115 120 200 220 230 240 250 260 is a flowchart of an example processassociated with managing request traffic in a serving system. In some implementations, one or more process blocks ofmay be performed by a device that includes offline systemand/or serving system. In some implementations, one or more process blocks ofmay be performed by another device or a group of devices separate from or including the device, such as data nodeand/or user device. Additionally, or alternatively, one or more process blocks ofmay be performed by one or more components of the device, such as processor, memory, input component, output component, and/or communication component.

310 300 At step, processmay include receiving, from a user device, a request. For example, the device may receive a request, as described herein. As an example, the request may indicate a search query.

320 300 At step, processmay include determining, based on the request, a retrieval set that includes a plurality of candidate results for the request. For example, the device may determine, based on the request, a retrieval set that includes a plurality of candidate results for the request, as described herein. The retrieval set may be a smaller subset of all possible candidate results, and may represent an initial set of candidate results for the request.

330 300 At step, processmay include generating a plurality of request-result pairs based on the request and the retrieval set. For example, the device may generate a plurality of request-result pairs based on the request and the retrieval set, as described herein. The request-result pairs provide the request and the candidate results of the retrieval set in a format that can be more easily processed by a model. In some implementations, the request-result pairs may be represented as embeddings.

340 300 At step, processmay include determining relevance outputs for the plurality of request-result pairs using a secondary model that is trained based on a primary model using: (1) an unlabeled dataset that is based on a candidate result set for a historical request, and (2) a margin-based loss function on all candidate result pairs of the candidate result set for the historical request. For example, the device may determine relevance outputs for the plurality of request-result pairs using a secondary model, as described herein. Based on the training using the unlabeled dataset and the margin-based loss function, the secondary model may have improved performance and may be capable of producing high-quality outputs.

350 300 At step, processmay include transmitting, to the user device and in accordance with the relevance outputs, data for presentation in a user interface of the user device based on the request. For example, the device may transmit, to the user device and in accordance with the relevance outputs, data for presentation in a user interface of the user device based on the request, as described herein. As an example, the data may include one or more candidate results (e.g., in accordance with the relevance outputs) formatted in HTML, or other markup or code.

3 FIG. 3 FIG. 300 300 300 Althoughshows example blocks of process, in some implementations, processmay include additional blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in. Additionally, or alternatively, two or more of the blocks of processmay be performed in parallel.

135 130 135 130 135 130 135 110 120 110 135 110 120 120 120 135 135 130 135 110 110 By training the secondary modelusing an unlabeled dataset (which can be much larger than the labeled dataset used to train the primary model) and the margin-based loss function, the secondary modelis able to achieve an efficacy that matches or exceeds that of the primary model. Because the secondary modelis much smaller than the primary model, but still as accurate, the secondary modelmay be deployed in the serving systemto handle request traffic from user deviceswhile satisfying the low-latency requirements of online, low-latency applications of the serving system. Moreover, by producing relevant and accurate outputs with low latency, the secondary modelenables the serving system, which is communicatively connected to user devicesvia a network, to respond to request traffic from the user devicesin real time with accurate, high-quality data that satisfies the user devices' requests. By doing so, fewer requests and responses are exchanged in network messages between the serving system and the user devices, thereby conserving significant processing resources and network resources. Furthermore, due to the way that the secondary modelis trained, the performance of the secondary modelis not affected even when fewer data fields (e.g., relative to the amount of data fields used in training of the primary model) are provided to the secondary modelas input. Reducing the number of data fields used as input significantly reduces the processing resources and power consumed by the serving system, while also reducing latency to achieve performance suitable for the online, low-latency applications of the serving system.

1. A method of managing request traffic in a serving system. 2. The method of aspect 1, comprising: receiving, by a device of the serving system and from a user device, a request; determining, by the device and based on the request, a retrieval set that includes a plurality of candidate results for the request; generating, by the device, a plurality of request-result pairs based on the request and the retrieval set; determining, by the device, relevance outputs for the plurality of request-result pairs using a secondary model that is trained based on a primary model using: (1) an unlabeled dataset that is based on a candidate result set for a historical request, and (2) a margin-based loss function on all candidate result pairs of the candidate result set for the historical request; and transmitting, by the device to the user device and in accordance with the relevance outputs, data for presentation in a user interface of the user device based on the request. 3. The method of any of aspects 1-2, wherein determining the relevance outputs for the plurality of request-result pairs using the secondary model comprises: inputting the plurality of request-result pairs into the secondary model, wherein a request-result pair, of the plurality of request-result pairs, comprises a combination of the request and a subset of a set of data fields used to train the primary model that is smaller in size than the set of data fields. 4. The method of any of aspects 1-3, further comprising: identifying, by the device, one or more frequently-used request-result pairs associated with the request; inputting, by the device, the one or more frequently-used request-result pairs into the secondary model to obtain frequently-used relevance outputs for the one or more frequently-used request-result pairs; storing, by the device, the frequently-used relevance outputs for the one or more frequently-used request-result pairs in a cache; and retrieving, by the device and based on the request, the frequently-used relevance outputs for the one or more frequently-used request-result pairs from the cache. 5. The method of aspect 4, wherein determining the relevance outputs for the plurality of request-result pairs using the secondary model comprises: identifying one or more request-result pairs of the plurality of request-result pairs that correspond to the one or more frequently-used request-result pairs; and determining a relevance ranking for the plurality of request-result pairs using the relevance outputs and the frequently-used relevance outputs. 6. The method of any of aspects 1-5, wherein the primary model and the secondary model have cross-encoder architectures. 7. The method of any of aspects 1-6, further comprising: training, by the device, the primary model, using a labeled dataset, to determine relevance between requests and results, wherein the labeled dataset includes a plurality of first request-result pairs and labels for the plurality of first request-result pairs; receiving, by the device, raw log data indicating the historical request and the candidate result set for the historical request; generating, by the device, an unlabeled dataset that includes a plurality of second request-result pairs based on the historical request and the candidate result set for the historical request; and training, by the device, the secondary model, based on knowledge distillation from the primary model, using the unlabeled dataset as input to the primary model and to the secondary model, and using the margin-based loss function on all candidate result pairs in the candidate result set for the historical request. 8. The method of aspect 7, wherein the margin-based loss function is a margin mean squared error (MSE) loss function and wherein training the secondary model comprises: training the secondary model using the margin MSE loss function for (1) a difference between first relevance outputs from the primary model for a first request-result pair and a second request-result pair, of the plurality of second request-result pairs, for the historical request, and (2) a difference between second relevance outputs of the secondary model for the first request-result pair and the second request-result pair for the historical request. 9. The method of any of aspects 7-8, wherein the first request-result pair and the second request-result pair relate to a pair of relevant candidate results to the historical request or to a pair of irrelevant candidate results to the historical request. 10. The method of any of aspects 7-9, wherein generating the unlabeled dataset comprises generating a first version of the unlabeled dataset that indicates the plurality of second request-result pairs using a set of data fields, and a second version of the unlabeled dataset that indicates the plurality of second request-result pairs using a subset of the set of data fields that is smaller in size than the set of data fields, and wherein training the secondary model comprises training the secondary model using the first version of the unlabeled dataset as input to the primary model and using the second version of the unlabeled dataset as input to the secondary model. 11. One or more non-transitory, computer-readable mediums storing instructions that, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising those of any of aspects 1-10. 12. A system comprising one or more processors; and memory storing instructions that, when executed by the processors, cause the processors to effectuate operations comprising those of any of aspects 1-10. 13. A system comprising means for performing any of aspects 1-10. The present techniques will be better understood with reference to the following enumerated aspects:

The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications may be made in light of the above disclosure or may be acquired from practice of the implementations.

Although particular combinations of features are recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the disclosure of various implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and/or disclosed in the specification.

No element, act, or instruction used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and may be used interchangeably with “one or more.” Further, as used herein, the article “the” is intended to include one or more items referenced in connection with the article “the” and may be used interchangeably with “the one or more.”