Enhancing Artificial Intelligence Responses with Contextual Usage Insights

Software applications offer a diverse array of features tailored to meet the needs of users across different domains. These features encompass functionalities ranging from basic operations to advanced capabilities, enhancing user experience and productivity. To aid users in navigating through these features effectively, documentation regarding the applications is often available. This documentation serves as a resource, elucidating the purpose, usage, and intricacies of each feature within the applications. Through user manuals, online guides, tutorials, tooltips, and the like, application documentation equips users with knowledge to more fully leverage the applications.

Some aspects of the present technology relate to, among other things, an artificial intelligence (AI) assistant system that generates responses to user queries regarding an application using documentation describing features of the application and usage data generated through use of the application. In accordance with some aspects, a user query is received by the AI assistant system. The user query is analyzed to determine a type of user question posed by the user query and processed with a corresponding pipeline to generate a response. In some aspects, the user query could comprise: a usage data question asking for certain usage data generated through use of the application, a concepts question asking for information regarding features of the application, or an out of scope question that does not ask for usage data or information regarding the application.

In instances in which the user query is determined to be a concepts question, the user query is processed by a concepts question pipeline of the AI assistant system. The concepts question pipeline leverages a generative model to generate text of a response to the concepts question using documentation describing the application. In some aspects, semantic similarity is used to retrieve, from a corpus of application documentation, documentation that is relevant to the user query, and the retrieved documentation is provided with the user query to the generative model for generating the response to the concepts question.

The AI assistant system also determines whether to add contextual usage data to the response. Contextual usage data comprises usage data that is helpful to provide context to the response to the concepts question generated from the application documentation. A usage insights pipeline determines usage data relevant to the user query and/or the response generated from the application documentation, queries a usage data store for that usage data, and leverages a generative model to generate text with the usage data to provide contextual usage insights. A final response is generated with the response to the concepts question and the contextual usage insights. The final response is provided to a user device for presentation as a response to the user query.

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Various terms are used throughout this description. Definitions of some terms are included below to provide a clearer understanding of the ideas disclosed herein.

As used herein, an “AI assistant” or “AI assistant system” refers to a computer-based system designed to generate responses to user queries through natural language interaction. An AI assistant can leverage artificial intelligence and machine learning algorithms to understand user queries, interpret context, and generate responses by accessing relevant information from various sources. In accordance with aspects of the technology described herein, an AI assistant generates responses to user queries regarding an application by leveraging usage data and/or application documentation for the application.

“Usage data” is used herein to refer to data collected through use of an application. It can include various information and metrics regarding user interactions with an application, such as, for instance, data pertaining to how enterprise users have utilized enterprise software. For example, when they create a segment, that segment “definition” is a piece of data maintained in servers/databases for the application. Similar usage data can be collected for other objects, such as schemas, sources, destinations, etc. As an example to illustrate in the context of web traffic monitoring, usage data may include the number of page views, unique visitors, session durations, referral sources, and popular content accessed on a website.

As used herein, “application documentation” or “documentation” is used to refer to information describing aspects of an application to support use, maintenance, or general understanding of the application. For instance, application documentation could include user manuals/guides, online help documentation, API documentation, technical documentation, release/version notes, community forums and support channels, knowledge bases, and training materials.

A “usage data question” refers to a user query in which a user is seeking to access particular usage data for an application. For example, a user could issue a user query such as: “how many page views were there on my website yesterday by males aged 30-40?” In accordance with some aspects of the technology described herein, a usage data response to a usage data question is generated using usage data generated for an application.

A “concepts question” refers to a user query in which a user is seeking general information regarding an application. For example, a user could issue a user query such as: “what is the difference between batch and streaming segmentation?” In accordance with some aspects of the technology described herein, a concepts response to a concepts question regarding an application is generated using documentation for the application.

An “out of scope question” refers to a user query that does not fall within the scope of a known type of user query, such as a usage data question or a concepts question.

“Contextual usage insights” is used herein to refer to usage data that is retrieved and added to a response to a concepts question to facilitate an understanding of the response.

While documentation for applications is undoubtedly valuable, users often encounter challenges when attempting to sift through extensive documentation to find answers. The sheer volume of information coupled with the need to interpret technical jargon can lead to frustration and inefficiency. However, advancements in conversational artificial intelligence (AI) have provided one way to address this issue. With the development of conversational AI assistants, users can ask natural language questions regarding applications and receive responses based on the documentation. These AI assistants leverage natural language processing (NLP) and machine learning algorithms to understand user queries, extract relevant information from documentation, and provide answers in real-time.

However, use of conversational AI assistants to answer application-related questions has essentially become a commoditized feature. In some cases, instead of using an in-house AI assistant, users could instead go directly to a general AI assistant, such as ChatGPT, to answer concepts questions. While the answers from AI assistants can be good enough, in some cases, the responses insufficiently answer the user questions. For instance, responses from currently-available AI assistants can be outdated, may be hard-to-fact check (i.e., hard to ensure the response from the AI assistant doesn't include any hallucinations), or may not clearly and adequately answer the user questions.

These shortcomings in existing AI assistant systems often result in the consumption of an unnecessary quantity of computing resources (e.g., I/O costs, network packet generation costs, throughput, memory consumption, etc.). For instance, if an AI assistant's response to a user query is insufficient, the user may need to submit additional queries to the AI assistant. In some instances, users may need to resort to searching the application documentation to find the information being sought when the AI assistant fails to provide adequate answers. These repetitive inputs result in increased computing resource consumption, among other things. For instance, repetitive user queries result in packet generation costs that adversely affect computer network communications. Each time a user issues a query, the contents or payload of the query is typically supplemented with header information or other metadata within a packet in TCP/IP and other protocols. Accordingly, when this functionality is multiplied by all the inputs needed to obtain the desired data, there are throughput and latency costs by repetitively generating this metadata and sending it over a computer network. In some instances, these repetitive inputs (e.g., user queries) increase storage device I/O (e.g., excess physical read/write head movements on non-volatile disk) because each time a user inputs unnecessary information, such as inputting several queries, the computing system often has to reach out to the storage device to perform a read or write operation, which is time consuming, error prone, and can eventually wear on components, such as a read/write head. Further, if users repetitively issue queries, it is expensive because processing queries consumes a lot of computing resources. This decreases throughput and increases network latency.

Aspects of the technology described herein improve the functioning of the computer itself in light of these shortcomings in existing technologies by providing a solution in which an AI assistant system not only answers a user's concepts question, but also provides contextual usage insights based on usage data generated from use of the application, thereby providing relevant context about how the application is being used. For example, suppose a user submits a concepts question to the AI assistant, such as “What is the difference between batch and streaming segmentation?” Aspects of the technology described herein generate a response to the user question by not only providing a technical answer based on application documentation, but also includes contextual usage data, such as stating how many batch and streaming segments have already been created in their usage data and some examples of each. These contextual usage insights provide valuable context to a user, helping ground the response in how the application is being used.

In accordance with some aspects of the present technology, when a user query is received, one or more query preprocessing operations may be performed, such as merging chat history from the current chat session with the user query and/or performing any number of query expansion operations (e.g., acronym expansion, synonym expansion, etc.). The user query may then be classified (e.g., by a classification model) as a particular type of user question and routed to an appropriate pipeline based on the type of user question presented. In some aspects, the AI assistant system can be configured to handle a number of different types of user questions. These may include, for instance, usage data questions in which users are seeking particular usage data for an application, concepts questions in which users are seeking general information regarding an application, and out of scope questions that does not fall within the scope of a known type of user query, such as a usage data question or a concepts question.

In the case of a concepts question, a concepts question pipeline processes the user query to generate a response based on application documentation. The concepts question pipeline uses a generative model to generate text to answer the concepts question using application documentation. In some aspects, documentation relevant to the user query is determined (e.g., using semantic similarity) from a corpus of documentation available for the application, and the retrieved documentation is provided to the generative model to generate the response to the concepts question. Source attribution can also be performed that identifies source documentation used to generate the response.

A determination is also made whether to add contextual usage insights to the response generated using the application documentation. In some configurations, a rules-based approach could be used to determine if the user query or the generated response contains known data objects for which usage data can be retrieved. In some aspects, the determination may also take into account whether a previous response during the chat session has already provided the contextual usage insights. When it is determined to add contextual usage insights to the concepts answer, a usage insights pipeline determines usage data is retrieved that is relevant to the user query and/or the response to the concepts question. A generative model then generates text with the retrieved usage data to generate contextual usage insights, which are merged with the response to the concepts question to generate a final response to the user query.

Aspects of the technology described herein provide a number of improvements over existing technologies. For instance, the technology described herein provides an AI assistant system that generates responses that enable enhanced user understanding. By integrating contextual usage insights into responses to concepts question, the AI assistant doesn't just provide textbook answers based on application documentation but offers responses based on data regarding actual usage of the application. This deepens the user's understanding, providing them with insights they might not even have realized they needed. As a result, an AI assistant system using the technology described herein provides improved computing resource consumption relative to existing technologies. For instance, aspects of the technology described herein eliminate (or at least reduce) the repetitive user queries relative to conventional techniques because the generated responses are more likely to fully address the user queries. Accordingly, aspects of the technology described herein decrease computing resource consumption, such as packet generation costs. For instance, a user query would only need to traverse a computer network once (or fewer times relative to existing technologies). Specifically, the contents or payload of the user query is supplemented with header information or other metadata within a packet in TCP/IP and other protocols once for the initial user query. A user query is only sent over the network once or fewer times. Thus, there is no repetitive generation of metadata and continuous sending of packets over a computer network. In like manner, aspects of the technology described herein improve storage device or disk I/O and query execution functionality, as they only need to go out to disk a single time (or fewer times relative to existing technologies). As described above, the inadequacy of existing approaches results in repetitive user queries, which causes multiple traversals to disk. In contrast, aspects described herein reduce storage device I/O because the user provides only minimal inputs and so the computing system does not have to reach out to the storage device as often to perform a read or write operation. Accordingly, there is not as much wear on components, such as a read/write head, because disk I/O is substantially reduced.

Example System for Enhancing AI Responses with Contextual Usage Insights

With reference now to the drawings,is a block diagram illustrating an exemplary systemproviding an AI assistant that generates responses to application-related user queries in accordance with implementations of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, and groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by one or more entities may be carried out by hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory.

The systemis an example of a suitable architecture for implementing certain aspects of the present disclosure. Among other components not shown, the systemincludes a user device, an application server, and an AI assistant system. Each of the user device, the application server, and the AI assistant systemshown incan comprise one or more computer devices, such as the computing deviceof, discussed below. As shown in, the user device, the application server, and the AI assistant systemcan communicate via a network, which may include, without limitation, one or more local area networks (LANs) and/or wide area networks (WANs). Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet. It should be understood that any number of user devices and servers may be employed within the systemwithin the scope of the present technology. Each may comprise a single device or multiple devices cooperating in a distributed environment. For instance, the application serverand the AI assistant systemcould each be provided by multiple server devices collectively providing the functionality of the application serverand the AI assistant systemas described herein. Additionally, other components not shown may also be included within the network environment.

The user devicecan be a client device on the client-side of operating environment, while the application serverand the AI assistant systemcan be on the server-side of operating environment. The application serverand/or the AI assistant systemcan each comprise server-side software designed to work in conjunction with client-side software on the user deviceso as to implement any combination of the features and functionalities discussed in the present disclosure. For instance, the user devicecan include an applicationfor interacting with the application serverand/or the AI assistant system. The applicationcan be, for instance, a web browser or a dedicated application for providing functions, such as those described herein. This division of operating environmentis provided to illustrate one example of a suitable environment, and there is no requirement for each implementation that any combination of the application serverand the AI assistant systemremain as separate entities. For instance, in some aspects, the AI assistant systemis a part of the application server. While the operating environmentillustrates a configuration in a networked environment with a separate user device, application server, and AI assistant system, it should be understood that other configurations can be employed in which aspects of the various components are combined.

The user devicemay comprise any type of computing device capable of use by a user. For example, in one aspect, a user device may be the type of computing devicedescribed in relation toherein. By way of example and not limitation, the user devicemay be embodied as a personal computer (PC), a laptop computer, a mobile or mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, global positioning system (GPS) or device, video player, handheld communications device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronic device, a workstation, or any combination of these delineated devices, or any other suitable device. A user may be associated with the user deviceand may interact with the application serverand/or the AI assistant systemvia the user device.

The application servercan be implemented using one or more server devices, one or more platforms with corresponding application programming interfaces, cloud infrastructure, and the like. The application serverprovides a framework for a web-based application that can be accessed by user devices, such as the user device, via the network. The application servercan support any type of application ranging from simple websites to complex enterprise applications, providing various features and functionalities such as data storage, user authentication, and real-time communication. The application serverstores usage data collected through use of the application in a usage data store. The usage data can comprise various information and metrics regarding user interactions with the application, such as, for instance, frequency and duration of usage, actions performed, features accessed, errors encountered, and user demographics. As an example to illustrate in the context of web traffic monitoring, usage data may include the number of page views, unique visitors, session durations, referral sources, and popular content accessed on a website. It should be noted that whileshows a configuration using a web-based application, in some aspects, the application can be provided (entirely or in part) on the user device.

The AI assistant systemprovides a conversational AI assistant that generates responses to user queries regarding the application provided by the application serverthrough natural language interaction. The AI assistant systemcan leverage artificial intelligence and machine learning algorithms to understand user queries, interpret context, and generate responses by accessing relevant information from various sources, including documentation regarding the application and usage data from the usage data store.

As shown in, the AI assistant systemincludes a query preprocessing module, a classification module, a usage data question module, an out of scope question module, and a concepts question module. The modules/components of the AI assistant systemmay be in addition to other components that provide further additional functions beyond the features described herein. The AI assistant systemcan be implemented using one or more server devices, one or more platforms with corresponding application programming interfaces, cloud infrastructure, and the like. While the AI assistant systemis shown separate from the application serverand the user devicein the configuration of, it should be understood that in other configurations, some of the functions of the AI assistant systemcan be provided on the application serverand/or the user device. Additionally, in some configurations, one or more of the components of the AI assistant systemshown incan be provided by the user device, the application server, and/or another location not shown in. The components can be provided by a single entity or multiple entities.

In some aspects, the functions performed by components of the AI assistant systemare associated with one or more applications, services, or routines. In particular, such applications, services, or routines may operate on one or more user devices, servers, may be distributed across one or more user devices and servers, or be implemented in the cloud. Moreover, in some aspects, these components of the AI assistant systemmay be distributed across a network, including one or more servers and client devices, in the cloud, and/or may reside on a user device. Moreover, these components, functions performed by these components, or services carried out by these components may be implemented at appropriate abstraction layer(s) such as the operating system layer, application layer, hardware layer, etc., of the computing system(s). Alternatively, or in addition, the functionality of these components and/or the aspects of the technology described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc. Additionally, although functionality is described herein with regards to specific components shown in example system, it is contemplated that in some aspects, functionality of these components can be shared or distributed across other components.

When the AI assistant systemreceives a user query from a user device, such as the user device, the query preprocessing moduleof the AI assistant systemcan perform one or more preprocessing operations on the user query to provide an enhanced query. The query preprocessing modulecan perform any of a variety of different types of query preprocessing. In some configurations, the query preprocessing moduleobtains the chat history from the current session and/or a previous session between the user and the AI assistant systemand merges the chat history with the user query. In some configurations, various query expansion operations can be performed for the user query. Query expansion generally comprises any technique used to reformulate the user query to improve performance of the AI assistant systemin generating a response to the user query. By way of example only and not limitation, one or more of the following query expansion processes can be performed on the user query:

User queries submitted to the AI assistant systemcan pose different types of user questions. In some instances, users issue user queries requesting information regarding their usage data, referred to herein as usage data questions. For example, a user could issue a user query such as: “how many page views were there on my website yesterday by males aged 30-40?” In other instances, users issue user queries seeking general information regarding the application, referred to herein as concepts questions. For example, a user could issue a user query such as: “what is the difference between batch and streaming segmentation?” Accordingly, the AI assistant systemincludes various modules for generating responses to different types of user questions. In the example of, the AI assistantincludes a usage data question modulefor handling usage data questions, a concepts question modulefor handling concepts questions, and an out of scope question modulefor handling other types of questions and user input. It should be understood that the modules,,are provided by way of example only, and the AI assistant systemcan include modules not shown for handling other specific types of user queries and/or omit some of the modules shown.

The AI assistant systemincludes a classification modulefor classifying a user query in order to select an appropriate module,,for generating a response to the user query. In the example of, the classification moduleclassifies a user query as a usage data question, a concepts question, or an out of scope question. However, it should be understood that in other configurations, the classification modulecan classify a user query as a different type of question or input.

In some configurations, the classification modulecomprises a machine learning model trained to classify a user query. By way of example only, the classification modulecan process a user query through an embedding model and a 2-layer neural network trained for 3-class classification. However, other model architectures could be employed. The machine learning model can be trained using a training dataset that includes labeled user queries—e.g., each user query has been labeled as a usage data question, a concepts question, or an out of scope question.

In accordance with aspects of the present technology, the concepts question moduleis configured to generate responses to concepts questions by not only providing general information regarding the application but also supplementing that information with usage data when appropriate to facilitate an understanding of the response. This usage data added to a response to a concepts question is referred to herein as contextual usage insights.

As shown in, the concepts question moduleincludes a concepts response componentand a contextual usage insights component. The concepts response componentgenerates a response that answers the concepts question using documentation regarding the application. There may be a variety of different sources of documentation providing information regarding an application. For instance, the documentation could include user manuals/guides, online help documentation, API documentation, technical documentation, release/version notes, community forums and support channels, knowledge bases, and training materials.

In some aspects, the concepts response componentemploys a generative model that, given a prompt based on the user query (or an enhanced query after query preprocessing), generates text that answers the concepts question based on the application documentation. The generative model can comprise a language model that includes a set of statistical or probabilistic functions to perform Natural Language Processing (NLP) in order to understand, learn, and/or generate human natural language content. For example, a language model can be a tool that determines the probability of a given sequence of words occurring in a sentence or natural language sequence. Simply put, it can be a model that is trained to predict the next word in a sentence. A language model is called a large language model (LLM) when it is trained on enormous amount of data and/or has a large number of parameters. Some examples of LLMs are GOOGLE's BERT and OpenAI's GPT-3 and GPT-4. These models have capabilities ranging from writing a simple essay to generating complex computer codes-all with limited to no supervision. Accordingly, an LLM can comprise a deep neural network that is very large (billions to hundreds of billions of parameters) and understands, processes, and produces human natural language by being trained on massive amounts of text. These models can predict future words in a sentence letting them generate sentences similar to how humans talk and write.

The generative model used by the concepts response componentcan comprise a neural network. As used herein, a neural network comprises multiple operational layers, including an input layer and an output layer, as well as any number of hidden layers between the input layer and the output layer. Each layer comprises neurons. Different types of layers and networks connect neurons in different ways. Neurons have weights, an activation function that defines the output of the neuron given an input (including the weights), and an output. The weights are the adjustable parameters that cause a network to produce a correct output.

As will be described in further detail below with reference to, in some aspects, the concepts response componentidentifies, from a corpus of application documentation, documentation that is relevant to the user query and provides that documentation to the generative model to generate the text for the response using the documentation. For instance, semantic similarity could be employed in which a query embedding is generated from the user query and used to identify similar document embeddings generated from the application documentation. In further aspects, the concepts response componentalso performs source attribution by identifying which application documentation was used to generate portions of the response and including references to the documentation in the response.

The contextual usage insights componentadds contextual usage data to responses generated by the concepts response component. In some aspects, the contextual usage insights componentinitially determines whether to add contextual usage insights. For instance, in some aspects, a rules-based approach is used in which the usage insights componentdetermines whether to add contextual usage insights based on whether the user query or the response generated by the concepts response componentincludes text corresponding with a data object for which usage data can be retrieved. Another rule may be based on whether contextual usage data (e.g., for that data object) has been provided in a previous response during the chat session. In further aspects, the contextual usage insights component employs a machine learning model trained to determine whether to add contextual usage insights to responses. The machine learning model could be trained on, for instance, user queries and/or responses labeled with an indication of whether to provide contextual usage insights.

Given a determination to add contextual usage insights, the contextual usage insights componentretrieves, from the usage data store, usage data relevant to the user query and/or the response generated by the concepts response component. As will be described in further detail below with reference to, in some aspects, the contextual usage insights componentmaintains a list of predefined context questions that relate to the usage data (e.g., “how many schemas are there in the usage data”; “what were themost recently created schemas”; etc.) mapped to queries in the appropriate syntax for retrieving data from the usage data store(e.g., SQL queries). In such configurations, semantic similarity can be used to identify one or more predefined context questions relevant to the user query and/or response, and the queries mapped to those predefined context question(s) can be used to retrieve the appropriate usage data.

The contextual usage insights componentleverages a generative model to generate natural language text with the retrieved usage data. In some aspects, the generative model generates text that is added as a separate section to the response generated by the concepts response componentto provide a final response. In other aspects, the text is interweaved with portions of the response from the concepts response componentto provide the final response.

With reference next to, a block diagram is provided showing an example processfor generating a response to a user query regarding an application in accordance with one aspect of the present technology. As shown at, a user queryis received. The user querymay be received as part of a chat session involving interaction between a user via a user device (such as the user deviceof) and an AI assistant (such as that provided by the AI assistant systemof).

One or more query preprocessing operationsare performed for the user queryto provide an enhanced query. The query preprocessing operationscan include, for instance, obtaining a chat history and merging the chat history with the user query. The chat history can comprise information regarding interaction between a user and the AI assistant from the current session, including user inputs and AI assistant responses. In some instances, the chat history can include information from previous sessions between the user and the AI assistant. The query preprocessing operationscan further include other query expansion operations, such as, for instance, acronym expansion, synonym expansion, stemming or lemmatization, word sense disambiguation, and the like.

The enhanced query is passed to a classification router, which selects a downstream pipeline for processing the enhanced query to generate a response. The classification routerselects the downstream pipeline based on the type of question involved. In the configuration of, the classification router identifies the type of question involved as a usage data question, a concepts question, or an out of scope question. The classification routecan correspond with the classification moduleof, and may comprise, for instance, a machine learning model (e.g., a neural network) trained to classify user queries as a usage data question, a concepts question, or an out of scope question.

Based on the classification determined by the classification router, the enhanced query is processed by a corresponding pipeline,, or, and a final responseis provided as output. For a usage data question, a usage data pipelineprocesses the enhanced query to generate a response that includes usage data from the application. For an out of scope question, an out of scope pipelineprocesses the enhanced query by to generate a response. For a concepts question, a concepts question pipelinegenerates a response based on information from documentation for the application. Additionally, a determinationis made regarding whether to inject contextual usage insights into the concepts response. If so, a usage insights pipelineenhances the concepts response from the concepts question pipelinewith contextual usage insights based on usage data from the application. If not, the response from the concepts response is not enhanced with contextual usage insights.

Turning next to, a block diagram is provided that illustrates an example processfor generating a response to a concepts question in accordance with some aspects of the technology described herein. As shown in, a user queryis initially processed by a concepts question pipelineto generate a concepts response that answers the concepts question of the user querybased on documentation for the application. The user querycould comprise an enhanced query after one or more query preprocessing operations. For example, the user querycould include a query input by a user enhanced with chat history from the user session, acronym expansion information, and/or other query preprocessing information. In some aspects, the user queryis determined to be a concepts question using a classification router, such as the classification routerof.

Given the user query, document retrievalis performed. In particular, the document retrievalidentifies application documentation that is relevant to the user queryand can be used for generating a concepts response to the user query. The identified documentation can comprise documents and/or document portions (e.g., snippets, phrases, sentences, paragraphs, sections, etc.) from the corpus of documentation available for the application.

In accordance with the configuration of, the document retrievalemploys semantic similarity to identify documents/document portions that are relevant to the user query. In particular, the user queryis provided to an embedding modelthat generates a query embedding. The query embedding is used to identify similar document embeddings from a document embedding database(e.g., using cosine similarity). The document embeddings databasestores document embeddings generated from documents/document portions from a source of application documentation. The document embeddings could be generated using the same embedding modelused to generate the query embedding.

In some aspects, the embedding modelused to generate query embeddings and/or document embeddings is a neural network model. The embedding modelcan a pre-trained model, such as a Bidirectional Encoder Representations from Transformers (BERT) model. In other instances, the embedding modelcan be built and trained from scratch or a pre-trained model that is fine-tuned for the semantic similarity task. For instance, the embedding modelcould be trained or fine-tuned on a training dataset in which example queries are paired with example documentation, such that for a given user query, the embedding modelcan retrieve semantically similar documentation.