Systems for determining representations of entities based on graph data using a language model

Machine learning models may be used to generate embeddings that are representative of an input.

While implementations are described in this disclosure by way of example, those skilled in the art will recognize that the implementations are not limited to the examples or figures described. It should be understood that the figures and detailed description thereto are not intended to limit implementations to the particular form disclosed but, on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope as defined by the appended claims. The headings used in this disclosure are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to) rather than the mandatory sense (i.e., meaning must). Similarly, the words “include”, “including”, and “includes” mean “including, but not limited to”.

An online store may offer a variety of items (e.g., physical goods, services, digital goods) for purchase, lease, subscription, and so forth. For example, an online store may include a collection of interfaces that present information regarding items, receive user input such as search queries or selection of navigational links, provide controls for initiating transactions and providing user input for other purposes, and so forth. In some cases, an interface may present information that is relevant to a user, such as recommendations or other information regarding items that are similar or related to items that a user has purchased, viewed, or otherwise interacted with. In other cases, information regarding items may be used to perform other types of tasks, such as price predictions or search query recommendations. Determining relationships and similarities between items and determining accurate information regarding items may represent a computationally intensive task that may be subject to inaccuracy. In some cases, information regarding various entities (e.g., items, item categories, brands or manufacturers of items, regions or geolocations where one or more items may be available, and search queries that have been received) may be stored as a graph. For example, graph data may include multiple nodes and multiple edges. Each node of the graph may represent characteristics of an entity, while each edge is associated with two nodes and represents a type of relationship between the entities represented by the nodes. For example, a node representing an item may include data indicative of characteristics of the item, a node representing an item category may include data indicative of characteristics of the category, a node representing a brand may include data indicative of characteristics of the brand, a node representing a region or geolocation may include data indicative of characteristics of the region or geolocation, and so forth. An edge connecting two nodes that represent items may indicate co-occurring user interactions for those items, while an edge connecting a node that represents a search query with a node that represents an item may indicate that receipt of the search query ultimately led to the purchase of the item. As other examples, edges that connect a node representing a brand or item category with a node representing an item may indicate that the item is included in that brand or category.

One method for determining items and other types of entities that are similar or related may include the generation of machine-readable representation data that represents characteristics of the entity. For example, based on input data representing the characteristics of an item, one or more machine learning models may determine representation data that includes a machine-readable representation of the characteristics of the item. One example type of representation includes a vector embedding, which may have multiple dimensions, each dimension corresponding to one or more characteristics, and the value associated with each dimension indicating the particular characteristic(s) of the entity represented by the embedding. The distance between two vector embeddings within a common embedding space may be used to determine a degree of similarity between the two entities represented by the embeddings. For example, a small distance between two vector embeddings may indicate that the represented entities are similar (e.g., having a significant number of identical or similar characteristics). Techniques to determine representation data for entities that accurately represent the characteristics of the entities may be used for various purposes, such as recommendations of items, item brands, or item categories, recommendations of search queries, predictions or prices or other item characteristics, and so forth. However, many types of machine learning models that are usable to determine such representations are not typically able to utilize the data and relationships represented in graphs, which may result in the determination of suboptimal representations or representations that include incomplete information.

Described in this disclosure are techniques that utilize a machine learning model, which in some implementations may include a large language model (LLM), to determine an entity representation indicative of characteristics associated with entities, based at least in part on graph data that represents characteristics of entities and types of relationships between entities. For example, graph data may include multiple nodes, each node representing characteristics of a particular entity (e.g., an item, item category, item brand, or search query). The graph data may also include a plurality of edges, each edge associated with two nodes and representing a type of relationship between the entities represented by the nodes. For example, one type of relationship represented by an edge may include the co-occurrence of user interactions with the associated entities. Continuing the example, if a threshold number of users that interact with a first item during a session also interact with a second item during the session, this relationship may be represented by an edge. Other example types of relationships represented by edges may include edges that associate items of a particular category with the node representing the category, items associated with a particular brand with the node representing the brand, and search queries that resulted in the purchase of an item with the item that was purchased.

For a selected entity, a relevant portion of the graph data (e.g., subgraph data) may be determined. In some implementations, the subgraph data may include the set of nodes and edges that are a threshold distance from the node representing the selected entity. For example, all nodes that are connected to the node representing the selected entity by an edge may be included in the subgraph data if the threshold distance is one. All nodes that are connected to a node that is connected to the node representing the selected entity may be included in the subgraph data if the threshold distance is two. A graph encoder or other type of machine learning model may be trained to determine a graph representation based on the subgraph data. The graph representation may represent the characteristics of the entity that are associated with the node that represents the entity, as well as at least a portion of the types of relationships represented by the edges that are connected to the node that represents the entity, and at least a portion of the characteristics of the other nodes associated with those edges. In some implementations, the graph encoder may be trained to predict occurrence of a first type of relationship—co-occurring user interactions during a session, while one or more other types of relationships are represented in the graph representation.

In some cases, a selected entity may also be associated with text data and image data, in addition to the data included in the graph. A text encoder or other type of machine learning model may be used to determine a text representation based on the text and semantic information of the text data. An image encoder or other type of machine learning model may be used to determine an image representation based on characteristics of the image data. The graph representation, text representation, and image representation may be used to determine an input to a machine learning model, which in some implementations may include an LLM, such as through use of initial tokens, separator tokens, tokens indicating the end of an input, and so forth. In some implementations, one or more of the graph representation, text representation, or image representation may be modified, such as through use of a projection technique, to map the representation to a dimensionality that is usable with an LLM or other type of model. The LLM or other type of model may be trained to determine entity representations indicative of characteristics of an entity based on one or more of graph data, text data, or image data, using training data that includes multiple pairs of values, such as seed values and target values. For example, the machine learning model may be trained to predict subsequent seed tokens, subsequent target tokens, target tokens that correspond to seed tokens, and contrastive loss associated with seed and target tokens. Continuing the example, an LLM or other type of machine learning model may be trained using training data that includes pairs of values, a first loss function based on prediction of a first value of a pair based on a second value of the pair, and a second loss function based on contrastive loss associated with the first value and the second value of the pair.

One or more outputs from the machine learning model may be used to determine an entity representation that represents characteristics of the entity. In some implementations, multiple outputs of the model may be processed using an averaging operation or a pooling operation to determine a final entity representation. At a subsequent time, the entity representations for one or more entities may be used for additional operations, such as item recommendations, search query recommendations, price predictions, predictions of other entity characteristics, and so forth.

Techniques described herein may therefore enable a machine learning model, which in some cases may include an LLM, to determine entity representations that accurately represent the characteristics of entities, and that incorporate the robust information included in graph data that may not necessarily be present in text data and image data that is specific to the entity. For example, an entity representation determined based in part on graph data may represent the types and characteristics of the relationships of the represented entity with other entities, and in some cases may also represent one or more characteristics of the related entities.

1 FIG. 1 FIG. 100 102 104 106 108 110 110 106 108 110 106 108 is a diagramdepicting an implementation of a system for determining entity representationsindicative of characteristics of an entity based on graph data, text data, and image dataassociated with the entity. As described previously, one example implementation of the system ofmay include use in conjunction with an online store from which one or more items (e.g., physical goods, digital goods, services) may be purchased, leased, subscribed, and so forth. As such, one example type of entity may include an item. For example, entity dataassociated with an item may be indicative of characteristics of the item, such as an item name, category, type or sub-type, price, size, color, materials, dimensions, ratings or reviews, and so forth. In some implementations, the entity datamay include the text data, such as text descriptive of the item, and image data, such as one or more images associated with the item. Other example types of entities may include item categories, or item brands or manufacturers, and the entity dataassociated with these entities may include text data, image data, or other types of data indicative of characteristics of the categories, brands, or manufacturers. Another example type of entity may include a search query. For example, characteristics of a search query may include the search terms, a time when the query was received, items that were included in an output responsive to the search query, user interactions such as purchases that occurred subsequent to presentation of the output, and so forth.

110 112 112 112 106 108 110 106 108 112 110 110 112 106 108 1 FIG. In some implementations, entity dataassociated with an entity may include an entity identifier. An entity identifiermay include any type of data that may be used to distinguish a particular entity from one or more other entities, such as a name, alphanumeric string, or other type of data. In some cases, an entity identifiermay include data that is not necessarily understandable to a human user but may be used by one or more computing devices to differentiate an entity from one or more other entities. Whiledepicts text dataand image dataincluded as entity data, in some implementations, text data, image data, or other data associated with an entity may be stored or otherwise maintained separate from the entity identifieror other entity data, and in response to receiving the entity dataor entity identifier, one or more computing devices may determine and access the text dataand image dataassociated with the entity.

1 FIG. 1 FIG. 114 102 104 106 108 114 114 depicts one or more representation serversthat may determine entity representationsbased on one or more of graph data, text data, or image data. Whiledepicts a single diagram of a representation server, in other implementations, the representation server(s)may include any number and any type of computing devices including, without limitation, one or more personal computing devices, portable computing devices, wearable computing devices, vehicle-based computing devices, servers, networked media devices, network-associated data storage devices, and so forth.

116 114 118 110 104 110 114 110 114 110 114 110 1 FIG. A subgraph moduleassociated with the representation server(s)may determine subgraph databased on the entity datafor an entity and the graph data. Whiledepicts the entity dataas a separate element from the representation server(s)for illustrative purposes, in some implementations, the entity datamay be stored in association with the representation server(s). In other implementations, the entity datamay be received from one or more other computing devices, such as through user input or an input from a service, machine learning model, or computing device. In still other implementations, input from a user or external device may be used to select or indicate an entity, and the representation server(s)may determine and access the entity dataassociated with the indicated entity.

104 104 106 108 110 104 110 104 110 110 104 104 110 104 110 104 110 104 The graph datamay represent characteristics of entities and relationships between entities. In some cases, one or more characteristics of an entity indicated in the graph datamay also be included in the text data, image data, or other entity data. For example, data that is associated with an entity represented in the graph datamay include text data, image data, or combinations thereof, that may be identical or similar to data included in the entity data. Continuing the example, a node representing an item may include text data, image data, or other data indicative of item characteristics, a node representing a brand may include text data, image data, or other data indicative of brand characteristics, a node representing an item category may include text data, image data, or other data indicative of category characteristics, a node representing a region may include text data, image data, or other data indicative of region characteristics, and so forth. In some cases, the graph datamay indicate characteristics of an entity that are not included in the entity data. For example, entity datamay not necessarily indicate relationships between the associated entity and other entities, or characteristics of other related entities, while the graph datamay include such information. Additionally, in some implementations, the graph datamay include types of data that are not included in the entity data. For example, the graph datamay include video data or other modalities of data that are not included in the entity data. As another example, the graph dataassociated with an entity may include information regarding the entity, such as counts of user interactions, transactions associated with the entity, and so forth, which may not necessarily be included in the entity data. In some implementations, the graph datamay include multiple nodes, each node representing a particular entity. Characteristics of the represented entity may be stored in association with each node. Relationships between entities may be represented by edges, each edge being associated with two nodes and indicating a relationship between the entities represented by each of the two nodes. As described previously, one example type of relationship represented by an edge may include the co-occurrence of user interactions with the associated entities. For example, if at least a threshold number of users that interact with a first item also interact with a second item, this relationship may be represented by an edge. Other example types of relationships represented by edges may include an indication of an association between an item and the item category to which the item belongs, or an indication of an association between an item and the brand or manufacturer of that item. Another example type of association may include a relationship between a search query that resulted in the purchase of an item with the item that was purchased. Other types of relationships may include an indication of items that are determined to be substitutes for one another, items that are identical but offered in different regions or using different interfaces, search queries that are identical but were used in different regions to search different item databases, and so forth.

118 104 112 104 116 120 118 104 110 120 118 118 118 116 118 120 104 118 120 118 116 118 The subgraph datamay be determined by determining a particular node of the graph datathat represents the entity associated with the entity identifier, then determining a subset of the nodes and edges of the graph datathat are associated with the node that represents the entity. In some implementations, the subgraph modulemay access threshold data, which may include one or more rules, algorithms, equations, threshold values, and so forth that may be used to determine the subgraph databased on the graph dataand entity data. For example, the threshold datamay indicate a threshold distance, relative to the node that represents the entity, that is to be used to determine the subgraph data. Continuing the example, if a threshold value indicates a distance of one, the subgraph datamay include the node that represents the entity and each node that is connected to the node that represents the entity by an edge. If a threshold value indicates a distance of two, the subgraph datamay also include each node connected by an edge to a node that is connected to the node that represents the entity. In other implementations, the subgraph modulemay determine the subgraph databased on one or more graph retrieval algorithms. For example, the threshold datamay include types of edges or other characteristics used to filter nodes and edges of the graph datato determine the subgraph data. As another example, the threshold datamay include one or more rules associated with business logic for an entity, and the subgraph datamay be determined based on entities and relationships between entities determined based on the business logic or associated rules. In still other implementations, the subgraph modulemay include one or more machine learning algorithms, and the subgraph datamay be determined based on one or more attention-based techniques or other types of trained models or algorithms.

122 114 124 118 124 118 118 124 A graph encoderassociated with the representation server(s)may determine a graph representationbased on the subgraph data. In some implementations, the graph representationmay include a vector embedding having dimensions that represent characteristics of subgraphs, and values for each dimension that represent the particular characteristics of the subgraph data. Because the subgraph dataincludes not only the node that represents the entity, but also one or more edges indicative of the type of relationship between the entity and other entities, and the nodes that represent one or more other entities, the graph representationmay represent the characteristics of the entity, as well as at least a portion of the types of relationships represented by the edges that are connected to the node that represents the entity, and at least a portion of the characteristics of the other nodes associated with those edges.

122 124 122 122 124 In some implementations, the graph encodermay be trained to determine graph representationsusing training data that includes multiple nodes indicative of entity characteristics, and edges indicative of types of relationships between entities. As described previously, one type of relationship represented by an edge may include co-occurrence of user interactions associated with two nodes. The graph encodermay be trained to predict occurrence of a first type of relationship using the training data and one or more loss functions. Other types of relationships may be used in message-passing between layers of the graph encoderand may be represented in the graph representation.

126 114 128 106 128 106 128 106 A text encoderassociated with the representation server(s)may determine a text representationbased on the text data. In some implementations, the text representationmay include a vector embedding having dimensions that represent characteristics of the text data, such as the inclusion of words, groups of words, sub-words, groups of sub-words, characters, groups of characters, and so forth. The text representationmay also represent semantic information associated with the text data, such as the order or arrangement of words or characters, proximity of words to other words, use of capitalization and punctuation, and so forth.

130 114 132 108 132 108 An image encoderassociated with the representation server(s)may determine an image representationbased on the image data. In some implementations, the image representationmay include a vector embedding having dimensions that represent characteristics of the image data, such as the locations and colors of pixels and so forth.

1 FIG. 122 126 130 118 106 108 106 108 118 118 108 106 118 106 108 Whiledescribes use of a graph encoder, text encoder, and image encoder, and the generation of vector embeddings as example types of representations, any type of machine learning model, network, algorithm, and so forth may be used to generate any type of representations associated with the subgraph data, text data, and image data. Additionally, in some implementations, one or more of the text dataor image datamay be absent or excluded from processing, and representations of only the subgraph dataor of only the subgraph dataand one of the image dataor text datamay be used. In some implementations, other types of data in addition to or in place of subgraph data, text data, or image datamay be associated with an entity. In such cases, encoders or other types of machine learning models may be used to determine representations based on the other type(s) of data.

134 114 136 124 128 132 134 138 136 136 124 128 132 136 124 128 132 136 124 128 132 136 An input moduleassociated with the representation server(s)may determine input databased on the graph representation, the text representation, and the image representation. In some implementations, the input modulemay access input parameters, which may include one or more rules, algorithms, threshold values, dimensionalities, processing steps, tokens, or other parameters that may be used to determine the input data. For example, generation of the input datamay include use of one or more projection operations to modify one or more of the graph representation, text representation, or image representationto correspond to the dimensions of an embedding space used by a large language model (LLM). In some cases, generation of the input datamay include the addition of one or more tokens, such as separator tokens, initial tokens, tokens indicating the end of an input, and so forth. In some implementations, multiple graph representations, text representations, or image representationsmay be determined, and generation of the input datamay include performing a pooling or averaging process to determine a final graph representation, text representation, or image representationfor use generating input data.

140 102 110 136 136 118 106 108 102 106 108 118 The LLMmay determine an entity representation, that represents the characteristics of the entity indicated in the entity data, based on the input data. Because the input datawas determined based on the subgraph data, text data, and image data, the entity representationmay represent not only the characteristics of the entity indicated in the text dataand image data, but also characteristics of the entity indicated in the subgraph data, characteristics of relationships between the entity and other entities, and characteristics of related entities.

140 102 104 106 108 140 140 102 140 102 140 1 FIG. In some implementations, the LLMmay be trained to determine entity representationsbased on one or more of graph data, text data, or image data, using training data that includes multiple pairs of values, such as seed values and target values. For example, the LLMmay be trained to predict subsequent seed tokens, subsequent target tokens, target tokens that correspond to seed tokens, and contrastive loss associated with seed and target tokens. Continuing the example, the LLMmay be trained using training data that includes pairs of values, a first loss function based on prediction of a first value of a pair based on a second value of the pair, and a second loss function based on contrastive loss associated with the first value and the second value of the pair. Whiledepicts generation of entity representationsusing an LLM, in other implementations, other types of machine learning models may be used. In some implementations an entity representationmay be determined based on one or more outputs associated with the LLM, such as by performing an averaging or pooling operation based on multiple outputs, by projecting or otherwise modifying or formatting the output(s) to a selected dimensionality or format, and so forth.

102 142 142 114 114 102 144 1 FIG. Entity representationsfor multiple entities may be generated and stored in a data repository. Whiledepicts a data repositoryas a separate element from the representation server(s)for illustrative purposes, in some implementations, the representation server(s)may store the entity representations. The entity representations may be used by one or more downstream devicesfor a variety of purposes.

102 144 146 For example, based on a distance or other metric for determining similarities between entities represented by entity representations, a downstream devicemay determine one or more outputsindicative of recommendations of items, recommendations of search queries or query terms, prediction of characteristics of items, and so forth.

2 FIG. 200 102 104 106 108 102 146 202 124 122 124 118 124 122 is a flow diagramdepicting an implementation of a method for determining entity representationsfor an entity based on graph data, text data, and image data, and using entity representationsto determine an output. At, a first machine learning model may be trained to determine graph representationsusing training data indicative of entity characteristics and relationships between entities. For example, the first machine learning model may include a graph encoderor another type of machine learning model that may generate graph representationsbased at least in part on subgraph data. The first machine learning model may be trained to predict a first type of relationship between entities, such as co-occurring user interactions, while other types of relationships may affect the determined graph representations. For example, a graph encodermay include many convolutional layers, and based on a loss function, may be trained to predict a first type of relationship while other types of relationships are used in message-passing between layers.

204 110 110 114 102 110 110 112 106 108 At, entity dataindicative of an entity may be determined. The entity datamay be determined based on user input, input from a service, machine learning model, or computing device, or automatically. For example, a representation serveror other computing device may be configured to automatically generate entity representationsbased on stored or accessible entity data. Entity dataassociated with an entity may include an entity identifierindicative of the entity and in some implementations may include one or more of text dataor image data. As described previously, in some implementations, types of entities may include items associated with an online store, search queries, item categories, or item brands or manufacturers.

206 118 110 104 104 118 120 118 118 1 FIG. At, subgraph datamay be determined based on correspondence between the entity dataand graph datathat is indicative of entity characteristics and relationships between entities. For example, in one implementation, the graph datamay include nodes that represent entities and may include information associated with entities, and edges that represent relationships between entities, each edge being associated with two nodes. One type of relationship represented by an edge may include co-occurrence of user interactions for two items represented by the associated nodes. Other example types of relationships represented by edges may include an indication of an association between an item and the item category to which the item belongs, or an indication of an association between an item and the brand or manufacturer of that item. Another example type of association may include a relationship between a search query that resulted in a purchase or other type of user interaction associated with an item. Other types of relationships may include an indication of items that are determined to be substitutes for one another, items that are identical or similar and offered in different regions or using different interfaces, search queries that are identical or similar and used in different regions to search different item databases, and so forth. As described with regard to, in some implementations, the subgraph datamay be determined based in part on threshold data, which may indicate a threshold distance value associated with a node that represents an entity. For example, based on a threshold distance value of one, subgraph datathat includes a node that represents the entity and each node connected to that node by an edge may be determined. Based on a threshold distance value of two, subgraph datathat also includes each node connected by an edge to a node that is connected to the node that represents the entity may be determined.

208 118 124 118 118 118 124 At, graph representation data may be determined based on the subgraph dataand the first machine learning model. For example, in one implementation, the graph representationmay include a vector embedding having dimensions that represent characteristics of the subgraph data, with the values for each dimension representing the particular characteristics of the subgraph data. Because the subgraph dataincludes the node that represents the entity, one or more edges indicative of relationships between the entity and other entities, and the nodes that represent one or more other entities, the graph representationmay represent the characteristics of the entity, the relationships, and at least a portion of the characteristics of the other entities.

210 106 108 106 108 110 106 108 110 106 108 At, text dataand image dataassociated with the entity may be determined. In some implementations, the text dataand image datamay be included in the entity data. In other implementations, the text dataand image datathat correspond to the entity indicated in the entity datamay be determined by accessing other data sources. In some cases, one or more of the text dataor image datamay be absent or excluded from use.

212 106 126 106 106 At, text representation data may be determined based on the text dataand a second machine learning model. For example, a text encoderor other type of machine learning model may be trained to determine representations of the words, subwords (e.g., portions of words), characters, and semantic information associated with text. The text representation data may therefore represent characteristics of the text data, such as the words included in the text data, and semantic information such as the arrangement of the words, the proximity of the words relative to other words, and so forth.

214 108 130 132 108 At, image representation data may be determined based on the image dataand a third machine learning model. For example, the third machine learning model may include an image encoderor another type of machine learning model. In some implementations, the image representationmay include a vector embedding having dimensions that represent characteristics of the image data, such as the locations and colors of pixels.

216 136 136 140 136 136 At, input datamay be determined based on the graph representation data, the text representation data, and the image representation data. In some implementations, the input datamay be determined by concatenating the graph representation data, text representation data, and image representation data, in some implementations with one or more initial tokens, separator tokens, or concluding tokens. Additionally, in some implementations, one or more of the graph representation data, text representation data, or image representation data may be modified, such as through use of a projection technique, to map the representation to a dimensionality that is usable with the LLM. Additionally, in some implementations, multiple graph, image, or text representations may be determined, and generation of the input datamay include use of an averaging or pooling technique to determine individual graph, image, or text representations for use in determining the input data.

218 102 140 140 At, the fourth machine learning model may be trained to determine entity representationsusing training data that includes pairs of values, a first loss function based on prediction of a first paired value based on a second paired value, and a second loss function based on contrastive loss associated with the first and second values. For example, in some implementations the machine learning model may include a large language model (LLM)that is provided with training data that includes paired seed and target values. The LLMmay be trained to predict subsequent seed tokens, subsequent target tokens, target tokens that correspond to seed tokens, and contrastive loss associated with seed and target tokens.

220 102 136 102 102 102 Atan entity representationmay be determined based on the input dataand the fourth machine learning model. In some implementations, the output from the fourth machine learning model may include the entity representation. In other implementations, the fourth machine learning model may determine multiple representations, and a final entity representationmay be determined using a pooling or averaging operation, or one or more other rules, algorithms, or thresholding operations. In some implementations, an output from the fourth machine learning model may be modified to determine the entity representation, such as through the addition or removal of data, use of a projection operation to modify the dimensionality of the output, and so forth.

222 146 102 102 102 102 102 144 146 102 102 146 102 102 102 144 144 102 114 144 2 FIG. 1 FIG. At, one or more outputsmay be determined based on a relationship between the entity representationand a second entity representationassociated with a second entity. For example, the process described with regard tomay be performed for multiple entities, such that each entity is associated with an entity representationthat represents the characteristics of that entity. Various operations, such as distance-determination operations within a common embedding space, may be used to determine similarities and differences between entities based on the entity representations. As described with regard to, entity representationsmay be stored and accessed by downstream devices, which may determine outputsbased on the entity representations. For example, similarities between entity representationsmay be used to determine identical, similar, related, or subsequent items, which may be used to determine outputsthat include item recommendations. As another example, similarities between entity representationsmay be used to recommend search queries or search query terms. Similarities and differences between entity representationsmay be used to predict characteristics of entities, classify or categorize entities, and so forth. Additionally, generation of entity representationsfor a set of entities that may be used by a large number of downstream devicesmay conserve time and computational resources by enabling each downstream deviceto be configured to utilize the same types of entity representations, while the process performed using the representation server(s)may be performed a single time, rather than by each downstream device.

3 3 FIGS.A andB 3 FIG.A 300 102 104 106 108 302 110 110 106 108 110 110 112 106 108 106 108 110 106 108 are a diagramdepicting an implementation of a method for determining entity representationsbased on graph data, text data, image data, and one or more machine learning models. As shown in, at, entity datamay be determined. The entity datamay include text dataand image dataassociated with an entity. For example, entity dataindicative of a particular entity may be received or indicated via user input, automated input associated with one or more services or computing devices, and so forth. The entity datamay include an entity identifierindicative of a particular entity, and in some implementations may also include text dataand image data. In other implementations, one or more of the text dataor the image datamay be determined from one or more other sources in response to determination of the entity data, receipt of input indicative of an entity, and so forth. In still other implementations, one or more of the text dataor image datamay be omitted.

304 118 110 104 104 104 120 110 118 110 At, subgraph datamay be determined based on the entity data, graph dataindicative of entity characteristics and relationships between entities, and a threshold distance value. For example, graph datamay include multiple nodes, each node representing an entity and characteristics of the represented entity. The graph datamay also include multiple edges, each edge associated with two nodes and representing a relationship between the entities represented by the nodes. Threshold datamay indicate one or more threshold distance values that represent a distance from a node that represents the entity indicated in the entity data. Subgraph datamay be determined based on a set of nodes and edges that are within the threshold distance of the node that represents the entity indicated in the entity data.

306 118 106 108 306 122 124 118 124 118 124 126 128 106 128 106 130 132 108 132 108 1 FIG. At, graph representation data may be determined based on the subgraph dataand a first machine learning model, text representation data may be determined based on the text dataand a second machine learning model, and image representation data may be determined based on the image dataand a third machine learning model. For example, the machine learning models described with regard tomay include encoders that generate representation data based on input data having a particular modality. Continuing the example, as described with regard to, a graph encodermay be used to determine a graph representationbased on the subgraph data. Because the graph representationis determined based on the subgraph data, the graph representationmay represent characteristics of not only the indicated entity, but also characteristics of the relationships between that entity and other entities, and characteristics of the related entities. A text encodermay be used to determine a text representationbased on the text data. The text representationmay represent the words and semantic information included in the text data. An image encodermay be used to determine an image representationbased on the image data. The image representationmay represent characteristics of the image(s) included in the image data.

3 FIG.B 308 136 138 138 136 136 124 128 132 140 136 124 128 132 136 124 128 132 136 As shown in, at, input datamay be determined based on the graph representation data, text representation data, image representation data, and input parameters. The input parametersmay include one or more rules, algorithms, threshold values, dimensionalities, processing steps, tokens, or other parameters that may be used to determine the input data. For example, generation of the input datamay include use of one or more projection operations to modify one or more of the graph representation, text representation, or image representationto correspond to the dimensions of an embedding space used by a subsequent machine learning model, such as a large language model (LLM). In some implementations, generation of the input datamay include the addition of one or more tokens, such as separator tokens, initial tokens, tokens indicating the end of an input, and so forth. In some implementations, multiple graph representations, text representations, or image representationsmay be determined, and generation of the input datamay include performing a pooling or averaging process to determine a final graph representation, text representation, or image representationfor use generating input data.

310 102 136 140 102 118 106 108 118 136 102 140 1 2 FIGS.and At, an entity representationmay be determined based on the input dataand a fourth machine learning model. As described with regard to, in some implementations, the fourth machine learning model may include an LLM. The entity representationmay represent the characteristics of the entity indicated in the subgraph data, text data, and image data, which due to the inclusion of information from the subgraph datain the input datamay represent not only the characteristics of the indicated entity, but also of relationships between the entity and other entities, and characteristics of related entities. In some implementations an entity representationmay be determined based on one or more outputs associated with the LLM, such as by performing an averaging or pooling operation based on multiple outputs, by projecting or otherwise modifying or formatting the output(s) to a selected dimensionality or format, and so forth.

312 102 102 142 144 142 At, the entity representationmay be stored for access by one or more computing devices. For example, entity representationsfor multiple entities may be generated and stored in a data repository. One or more downstream devicesmay access the data repositoryto perform various operations, such as recommendations of similar or related items, brands, categories, or search queries, prediction of characteristics of items, and so forth.

4 FIG. 1 FIG. 4 FIG. 400 402 402 114 144 402 400 402 402 116 122 126 130 140 140 116 122 126 130 102 142 102 142 102 144 is a block diagramdepicting an implementation of a computing devicewithin the present disclosure. The computing devicemay include one or more representation servers, as described with regard to. In other implementations, one or more downstream devices, or other computing devices in communication with the computing device, may perform one or more of the functions described herein. Therefore, whiledepicts a single block diagramrepresentative of a computing device, any number of computing devicesmay be used, of similar or differing types. For example, a first computing device or set of computing devices may be used to train one or more of the subgraph module, graph encoder, text encoder, image encoder, or LLM, while a second computing device or set of computing devices may store and execute the LLMand in some cases one or more of the subgraph module, graph encoder, text encoder, or image encoder. As another example, a first computing device or set of computing devices may determine entity representations, while a second computing device or set of computing devices may include one or more data repositoriesfor storing entity representations. In some cases, a data repositorymay include a cache or other type(s) of memory that may enable entity representationsto be accessed efficiently for one or more uses by downstream devices.

404 402 404 One or more power suppliesmay be configured to provide electrical power suitable for operating the components of the computing device. In some implementations, the power supplymay include a rechargeable battery, fuel cell, photovoltaic cell, power conditioning circuitry, and so forth.

402 406 406 408 406 408 The computing devicemay include one or more hardware processor(s)(processors) configured to execute one or more stored instructions. The processor(s)may include one or more cores. One or more clock(s)may provide information indicative of date, time, ticks, and so forth. For example, the processor(s)may use data from the clockto generate a timestamp, trigger a preprogrammed action, and so forth.

402 410 412 414 410 402 402 402 402 412 The computing devicemay include one or more communication interfaces, such as input/output (I/O) interfaces, network interfaces, and so forth. The communication interfacesmay enable the computing device, or components of the computing device, to communicate with other computing devicesor components of the other computing devices. The I/O interfacesmay include interfaces such as Inter-Integrated Circuit (I2C), Serial Peripheral Interface bus (SPI), Universal Serial Bus (USB) as promulgated by the USB Implementers Forum, RS-232, and so forth.

412 416 416 402 416 416 402 416 The I/O interface(s)may couple to one or more I/O devices. The I/O devicesmay include any manner of input devices or output devices associated with the computing device. For example, I/O devicesmay include touch sensors, keyboards, mouse devices, microphones, image sensors, cameras, scanners, displays, speakers, haptic devices, printers, and so forth. In some implementations, the I/O devicesmay be physically incorporated with the computing device. In other implementations, I/O devicesmay be externally placed.

414 402 416 414 414 The network interfacesmay be configured to provide communications between the computing deviceand other devices, such as the I/O devices, routers, access points, and so forth. The network interfacesmay include devices configured to couple to one or more networks including local area networks (LANs), wireless LANs (WLANs), wide area networks (WANs), wireless WANs, and so forth. For example, the network interfacesmay include devices compatible with Ethernet, Wi-Fi, Bluetooth, ZigBee, Z-Wave, 5G, LTE, and so forth.

402 402 The computing devicemay include one or more buses or other internal communications hardware or software that allows for the transfer of data between the various modules and components of the computing device.

4 FIG. 402 418 418 418 402 418 As shown in, the computing devicemay include one or more memories. The memorymay include one or more computer-readable storage media (CRSM). The CRSM may be any one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, a mechanical computer storage medium, and so forth. The memorymay provide storage of computer-readable instructions, data structures, program modules, and other data for the operation of the computing device. A few example modules are shown stored in the memory, although the same functionality may alternatively be implemented in hardware, firmware, or as a system on a chip (SoC).

418 420 420 412 414 416 406 420 The memorymay include one or more operating system (OS) modules. The OS modulemay be configured to manage hardware resource devices such as the I/O interfaces, the network interfaces, the I/O devices, and to provide various services to applications or modules executing on the processors. The OS modulemay implement a variant of the FreeBSD operating system as promulgated by the FreeBSD Project; UNIX or a UNIX-like operating system; a variation of the Linux operating system as promulgated by Linus Torvalds; the Windows operating system from Microsoft Corporation of Redmond, Washington, USA; or other operating systems.

422 418 422 422 422 402 One or more data storesand one or more of the following modules may also be associated with the memory. The modules may be executed as foreground applications, background tasks, daemons, and so forth. The data store(s)may use a flat file, database, linked list, tree, executable code, script, or other data structure to store information. In some implementations, the data store(s)or a portion of the data store(s)may be distributed across one or more other devices including other computing devices, network attached storage devices, and so forth.

424 402 A communication modulemay be configured to establish communications with one or more other computing devices. Communications may be authenticated, encrypted, and so forth.

418 116 116 118 110 104 120 104 110 120 118 104 110 110 118 104 110 104 120 116 120 104 118 120 118 104 116 118 The memorymay store the subgraph module. The subgraph modulemay determine subgraph databased on entity data, graph data, and threshold data. The graph datamay represent characteristics of entities and relationships between entities. The entity datamay indicate a particular entity. The threshold datamay indicate one or more rules, algorithms, equations, threshold values, and so forth that may be used to determine the subgraph databased on the graph dataand entity data, such as a threshold distance value from a node that represents the entity indicated in the entity data. For example, the subgraph datamay be determined by determining a particular node of the graph datathat represents the entity indicated in the entity data, then determining a subset of the nodes and edges of the graph datathat are associated with the node that represents the entity based on the threshold data. In some implementations, the subgraph modulemay utilize one or more graph retrieval algorithms. For example, the threshold datamay indicate one or more characteristics of nodes or edges that may be used to filter the graph datato determine the subgraph data. As another example, the threshold datamay include one or more rules associated with business logic for an entity, and the subgraph datamay be determined based on entities and relationships between entities represented in the graph dataand indicated in the business logic or associated rules. In other implementations, the subgraph modulemay include one or more machine learning algorithms, and the subgraph datamay be determined based on one or more attention-based techniques or other types of models or algorithms.

418 122 122 124 118 124 118 118 124 122 124 122 122 124 The memorymay also store the graph encoder. The graph encodermay determine graph representationsbased on subgraph data. In some implementations, a graph representationmay include a vector embedding having dimensions and values that represent characteristics of the represented subgraph data. Because the subgraph dataincludes the node that represents the entity, one or more edges indicative of relationships between the entity and other entities, and one or more nodes that represent related entities, the graph representationmay represent the characteristics of the entity, as well as at least a portion of the relationships represented by the edges and the characteristics of the related entities. In some implementations, the graph encodermay be trained to determine graph representationsusing training data that includes multiple nodes indicative of entity characteristics, and edges indicative of relationships between entities. For example, one type of relationship represented by an edge may include co-occurrence of user interactions associated with items represented by two nodes. The graph encodermay be trained to predict occurrence of a first type of relationship using the training data and one or more loss functions. Other types of relationships may be used in message-passing between layers of the graph encoderand may be represented in the graph representation.

418 126 126 128 106 128 106 128 106 The memorymay additionally store the text encoder. The text encodermay determine text representationsbased on text dataassociated with an entity. In some implementations, the text representationmay include a vector embedding having dimensions that represent characteristics of the text data, such as the inclusion of words, groups of words, sub-words, groups of sub-words, characters, groups of characters, and so forth. The text representationmay also represent semantic information associated with the text data, such as the order or arrangement of words or characters, proximity of words to other words, use of capitalization and punctuation, and so forth.

418 130 130 132 108 108 108 132 108 The memorymay store the image encoder. The image encodermay determine image representationsbased on image dataassociated with an entity. For example, image datamay include one or more images depicting an item, portions of an item, various views of the item, and so forth. In some cases, image datamay include a logo associated with a brand or manufacturer, a symbol or image associated with a category of items, and so forth. In some implementations, the image representationmay include a vector embedding having dimensions that represent characteristics of the image data, such as the locations and colors of pixels and so forth.

418 134 134 136 124 128 132 134 138 136 124 128 132 140 124 128 132 136 124 128 132 136 136 124 128 132 The memorymay also store the input module. The input modulemay determine input databased on one or more graph representations, text representations, and images representation. In some implementations, the input modulemay access input parameters, which may include one or more rules, algorithms, threshold values, dimensionalities, processing steps, tokens, or other parameters that may be used to determine the input data. For example, in some cases, one or more projection operations may be used to modify the dimensionality of one or more of a graph representation, text representation, or image representationto correspond to an embedding space used by an LLMor other type of machine learning model. In some cases, multiple graph representations, text representations, or image representationsmay be determined, and generation of the input datamay include performing a pooling or averaging process to determine a final graph representation, text representation, or image representationfor use generating input data. In some cases, generation of the input datamay include concatenation of a graph representation, text representation, and image representation, the addition of one or more tokens, such as initial tokens, separator tokens, and concluding tokens, and so forth.

418 140 140 102 136 136 118 106 108 102 106 108 118 140 140 The memorymay additionally store the large language model (LLM)or one or more other types of machine learning models. The LLMmay be trained to determine entity representationsbased on input data. Because the input datawas determined based on subgraph data, text data, and image data, the entity representationmay represent the characteristics of the entity indicated in the text dataand image data, as well as relationships between the entity and other entities, and characteristics of related entities, indicated in the subgraph data. In some implementations, the LLMmay be trained using training data that includes multiple pairs of values, such as seed values and target values. For example, the LLMmay be trained to predict subsequent seed tokens, subsequent target tokens, target tokens that correspond to seed tokens, and contrastive loss associated with seed and target tokens.

426 418 402 402 426 104 120 138 426 Other modulesmay also be present in the memory. For example, encryption modules may be used to encrypt and decrypt communications between computing devices. Authentication modules may be used to authenticate communications sent or received by computing devices. Other modulesmay also include modules for receiving user input, generating user interfaces such as interfaces that present output, or for modifying graph data, threshold data, input parameters, and so forth. Other modulesmay include modules for training and tuning encoders and machine learning models, modifying parameters associated with encoders and machine learning models, and so forth.

428 422 402 428 428 140 428 428 Other datawithin the data store(s)may include configurations, settings, preferences, and default values associated with computing devices. Other datamay also include encryption keys and schema, access credentials, and so forth. Other datamay include corpus text for use with the LLM. Other datamay further include user interface data for receiving queries and other input and for presenting output. Other datamay also include training data for training of the encoders and machine learning models, loss functions associated with the encoders and machine learning models, and so forth.

402 114 144 In different implementations, different computing devicesmay have different capabilities or capacities. For example, representation serversthat store and execute components of the system may have greater processing capabilities or data storage capacity than downstream devices.

The processes discussed in this disclosure may be implemented in hardware, software, or a combination thereof. In the context of software, the described operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more hardware processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. Those having ordinary skill in the art will readily recognize that certain steps or operations illustrated in the figures above may be eliminated, combined, or performed in an alternate order. Any steps or operations may be performed serially or in parallel. Furthermore, the order in which the operations are described is not intended to be construed as a limitation.

Embodiments may be provided as a software program or computer program product including a non-transitory computer-readable storage medium having stored thereon instructions (in compressed or uncompressed form) that may be used to program a computer (or other electronic device) to perform processes or methods described in this disclosure. The computer-readable storage medium may be one or more of an electronic storage medium, a magnetic storage medium, an optical storage medium, a quantum storage medium, and so forth. For example, the computer-readable storage media may include, but is not limited to, hard drives, optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable ROMs (EPROMs), electrically erasable programmable ROMs (EEPROMs), flash memory, magnetic or optical cards, solid-state memory devices, or other types of physical media suitable for storing electronic instructions. Further, embodiments may also be provided as a computer program product including a transitory machine-readable signal (in compressed or uncompressed form). Examples of transitory machine-readable signals, whether modulated using a carrier or unmodulated, include, but are not limited to, signals that a computer system or machine hosting or running a computer program can be configured to access, including signals transferred by one or more networks. For example, the transitory machine-readable signal may comprise transmission of software by the Internet.

Separate instances of these programs can be executed on or distributed across any number of separate computer systems. Although certain steps have been described as being performed by certain devices, software programs, processes, or entities, this need not be the case, and a variety of alternative implementations will be understood by those having ordinary skill in the art.

Additionally, those having ordinary skill in the art will readily recognize that the techniques described above can be utilized in a variety of devices, environments, and situations. Although the subject matter has been described in language specific to structural features or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claims.