A method of extending a conversational computing interface. The method comprises executing a nonnative skill implemented in a nonnative programming language of the conversational computing interface. The method further comprises automatically computer-tracing computer operations performed by the nonnative skill during such execution. The method further comprises automatically computer-generating a native computer-executable plan representing the traced computer operations in a native programming language of the conversational computing interface.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
2. The method of claim 1, wherein automatically computer-tracing the computer operations performed by the nonnative skill includes automatically computer-tracing the computer operations via a coarse grain tracing approach that automatically records when the nonnative skill calls a function of a predefined plurality of functions, and automatically records results of calling the function.
This invention relates to computer-based systems for tracing and analyzing nonnative skills, which are specialized software components or functions that are not natively integrated into a primary software application. The problem addressed is the difficulty in monitoring and understanding the behavior of these nonnative skills, particularly their interactions with the host system and the outcomes of their operations. The method involves automatically tracing computer operations performed by nonnative skills using a coarse-grain tracing approach. This approach records when the nonnative skill invokes any function from a predefined set of functions, capturing the timing and context of these calls. Additionally, the method records the results of these function calls, providing a high-level overview of the skill's behavior without requiring detailed, low-level tracing. This allows for efficient monitoring while still capturing critical operational data. The predefined plurality of functions represents key operations that are relevant to the nonnative skill's functionality, ensuring that only meaningful interactions are recorded. By focusing on these specific functions, the system avoids excessive data collection, reducing computational overhead while still providing sufficient insight into the skill's performance. This approach is particularly useful in environments where nonnative skills interact with multiple systems or perform complex tasks, as it simplifies the analysis of their behavior.
3. The method of claim 1, wherein automatically computer-tracing the computer operations performed by the nonnative skill includes automatically computer-tracing the computer operations via a fine grain tracing approach that automatically records computational events in an execution environment associated with the nonnative skill.
This invention relates to a system for tracing and analyzing computer operations performed by a nonnative skill, such as a software component or process that operates outside a user's primary expertise. The problem addressed is the difficulty in monitoring and understanding the computational behavior of such nonnative skills, which can lead to inefficiencies, errors, or security vulnerabilities. The method involves automatically tracing the computer operations of the nonnative skill using a fine-grained tracing approach. This approach records detailed computational events within the execution environment of the nonnative skill. The tracing captures low-level operations, such as function calls, data accesses, and system interactions, providing a comprehensive record of the skill's behavior. This detailed tracing allows for in-depth analysis, debugging, and optimization of the nonnative skill's performance. The fine-grained tracing is performed without manual intervention, ensuring continuous and accurate monitoring. The recorded events can be used to identify performance bottlenecks, security risks, or unexpected behavior. The method supports real-time or post-execution analysis, enabling developers to improve the reliability and efficiency of nonnative skills. This approach is particularly useful in complex systems where multiple nonnative skills interact, ensuring transparency and control over their operations.
4. The method of claim 3, wherein the nonnative skill includes a program configured for execution in an execution environment defined by a virtual machine, and wherein the fine grain tracing approach is implemented via configuring the virtual machine to automatically record each instruction executed by the virtual machine.
This invention relates to a system for monitoring and analyzing nonnative skills, particularly those involving programs executed in a virtual machine environment. The problem addressed is the difficulty in tracking and understanding the behavior of such programs, especially when they involve complex or nonstandard operations that are not easily observable through conventional debugging or profiling tools. The solution involves a fine-grained tracing approach that leverages the virtual machine itself to record detailed execution data. Specifically, the virtual machine is configured to automatically log each instruction it executes, providing a comprehensive trace of the program's behavior. This allows for deep inspection of the program's operations, including interactions with the virtual machine's execution environment, which may not be visible through traditional monitoring methods. The system is designed to work with programs that are not natively supported by the virtual machine, meaning they may rely on custom or nonstandard features that require specialized tracing mechanisms. By capturing instruction-level data, the system enables detailed analysis of how these programs operate, including their performance characteristics, resource usage, and potential errors or inefficiencies. The recorded traces can be used for various purposes, such as debugging, performance optimization, security analysis, or compliance monitoring. The fine-grained nature of the data ensures that even subtle or intermittent issues can be identified and addressed. The approach is particularly useful in environments where programs must interact with the virtual machine in nonstandard ways, such as in specialized applications or research settings.
5. The method of claim 3, wherein the nonnative skill includes a program configured for execution by compiling the program to machine instructions and executing the machine instructions in an execution environment associated with a computer processor, and wherein the fine grain tracing approach includes automatically recording each machine instruction executed by the computer processor.
This invention relates to a system for tracing and analyzing nonnative skills, particularly those involving program execution. The technology addresses the challenge of monitoring and debugging programs at a fine-grained level, where traditional tracing methods may lack precision or efficiency. The method involves a program configured for execution, which is compiled into machine instructions and then executed in an environment associated with a computer processor. A fine-grain tracing approach is employed to automatically record each machine instruction executed by the processor. This detailed tracing allows for precise tracking of program behavior, including instruction-level execution flow, which is useful for debugging, performance optimization, and security analysis. The tracing mechanism captures every machine instruction, providing a comprehensive record of execution. This level of detail enables deep analysis of program behavior, including identifying inefficiencies, detecting anomalies, or verifying correctness. The system may be applied to various computing environments, including virtual machines, emulators, or hardware-assisted tracing systems, where fine-grained instruction-level monitoring is required. By recording each machine instruction, the method ensures that no execution detail is overlooked, making it particularly valuable for complex or security-sensitive applications. The approach may also support dynamic analysis, where tracing data is used to guide further execution or optimization decisions. Overall, the invention provides a robust solution for instruction-level program tracing, enhancing debugging and analysis capabilities in computing systems.
6. The method of claim 1, wherein automatically computer-tracing the computer operations performed by the nonnative skill includes automatically computer-tracing the computer operations via a hybrid tracing approach that automatically records when the nonnative skill calls a function of a predefined plurality of functions via a coarse grain tracing approach, and further automatically records computational events associated with calling any other function not among the predefined plurality of functions via a fine grain tracing approach.
This invention relates to computer operation tracing for nonnative skills, addressing the challenge of efficiently monitoring and analyzing computational events in systems where certain skills or modules are not natively integrated. The method employs a hybrid tracing approach to balance performance and granularity. It automatically records when a nonnative skill invokes a function from a predefined set of critical functions using a coarse-grained tracing method, which captures high-level interactions with minimal overhead. For all other function calls not in the predefined set, the method switches to a fine-grained tracing approach, recording detailed computational events such as execution paths, memory access, and system calls. This dual-mode tracing ensures comprehensive monitoring while optimizing resource usage by avoiding excessive fine-grained tracing for non-critical operations. The hybrid approach dynamically adapts to the operational context, improving efficiency and accuracy in diagnosing performance issues, debugging, or security analysis. The invention is particularly useful in environments where nonnative skills interact with host systems, such as plugins, extensions, or third-party integrations, where understanding their behavior is essential for system stability and security.
7. The method of claim 1, further comprising extending a nonnative programming language by providing access to a native program function of the native programming language of the conversational computing interface, wherein the nonnative skill is implemented in the nonnative programming language using the native program function of the native programming language.
This invention relates to extending a nonnative programming language within a conversational computing interface by enabling access to native program functions of the interface's native programming language. The system allows developers to implement nonnative skills or functionalities using the nonnative programming language while leveraging the capabilities of the native programming language. The conversational computing interface processes user inputs, such as voice or text commands, and executes tasks based on the implemented skills. The extension mechanism bridges the nonnative and native programming languages, allowing the nonnative language to call and utilize native functions, thereby enhancing the functionality of the nonnative skills. This approach enables developers to build and integrate custom skills into the conversational interface without being limited to the native programming language, improving flexibility and ease of development. The system ensures seamless interaction between the nonnative and native components, maintaining the interface's ability to process and respond to user inputs effectively.
8. The method of claim 7, wherein the native program function is executable to access a conversation history of past interactions with the conversational computing assistant.
This invention relates to conversational computing assistants and methods for enhancing their functionality by integrating native program functions. The problem addressed is the limited ability of conversational assistants to leverage past interactions when executing tasks, leading to inefficiencies and repetitive user inputs. The solution involves a method where a native program function, executed by the assistant, accesses a conversation history of past interactions to improve task performance. The native program function is designed to retrieve and utilize relevant data from prior conversations, such as user preferences, context, or previous commands, to streamline current operations. This integration allows the assistant to maintain context across interactions, reducing the need for redundant user inputs and enhancing the overall user experience. The method ensures that the assistant can dynamically adapt its responses and actions based on historical data, making interactions more efficient and personalized. By accessing conversation history, the assistant can also identify patterns or trends in user behavior, further optimizing its performance over time. This approach improves the assistant's ability to handle complex, multi-step tasks by referencing past interactions, thereby providing a more seamless and intuitive user experience.
9. The method of claim 7, wherein the native program function is a synchronous native program function configured to return a result value to the nonnative program before continuing execution of the nonnative program.
A system and method for interfacing non-native programs with native program functions, particularly addressing the challenge of integrating non-native programs with native functions that require synchronous execution. The method involves executing a non-native program that includes a call to a native program function. The native function is designed to operate synchronously, meaning it must return a result value to the non-native program before the non-native program can continue execution. The system ensures that the non-native program waits for the native function to complete and return the result before proceeding. This approach prevents execution errors that could arise from the non-native program attempting to use the result of the native function before it is available. The method may also include handling errors that occur during the execution of the native function, such as by propagating the error back to the non-native program or taking corrective actions. The system may further include mechanisms for managing the execution context, such as passing parameters between the non-native and native programs and ensuring proper synchronization of execution states. This solution is particularly useful in environments where non-native programs must interact with native functions that require synchronous behavior, such as in operating system calls or hardware-specific operations.
10. The method of claim 7, wherein the native program function is an asynchronous native program function configured to asynchronously yield control to the nonnative program, and to asynchronously supply a result value to the nonnative program via an asynchronous programming interface of the nonnative program.
This invention relates to asynchronous programming interfaces in computing systems, specifically addressing the challenge of integrating native program functions with non-native programs in a way that allows for asynchronous execution and result handling. The method involves using an asynchronous native program function that is designed to temporarily yield control to a non-native program while the native function performs its operations. Once the native function completes, it asynchronously supplies a result value back to the non-native program through an asynchronous programming interface. This approach enables efficient and non-blocking interaction between different types of programs, improving performance and responsiveness in systems where native and non-native components must work together. The asynchronous programming interface of the non-native program is used to manage the flow of control and data, ensuring that the non-native program can continue executing other tasks while waiting for the native function to complete. This method is particularly useful in environments where native code must be called from higher-level or managed code, such as in embedded systems, operating system development, or cross-platform applications. The solution avoids the inefficiencies of synchronous calls, which would otherwise block the non-native program until the native function finishes.
11. The method of claim 10, wherein the asynchronous native program function is an input function configured to receive an utterance from a user.
This invention relates to asynchronous native program functions in computing systems, specifically focusing on input functions that receive user utterances. The technology addresses the challenge of efficiently handling asynchronous operations in software applications, particularly when processing user inputs like voice commands or spoken language. The method involves executing an asynchronous native program function that is specifically designed to capture and process user utterances, such as spoken words or phrases, without blocking the main program flow. This allows the system to remain responsive while handling input operations in the background. The asynchronous function may include mechanisms to parse, interpret, or transmit the received utterance for further processing, such as natural language understanding or command execution. The invention ensures that input functions operate independently of the main program thread, improving system performance and user experience by avoiding delays or interruptions during input handling. This approach is particularly useful in applications requiring real-time interaction, such as voice assistants, virtual agents, or interactive software interfaces. The method may also include error handling and synchronization features to manage asynchronous operations reliably.
12. The method of claim 1, further comprising automatically machine-learning training the conversational computing interface based on the native computer-executable plan and a conversational event associated with the native computer-executable plan.
This invention relates to conversational computing interfaces, specifically improving their training through machine learning. The problem addressed is the need for automated, adaptive training of conversational interfaces to enhance their performance in executing computer-executable plans. A conversational computing interface is a system that processes natural language inputs to perform tasks defined by computer-executable plans, which are structured sets of instructions for achieving specific outcomes. The method involves automatically training the conversational interface using machine learning. The training is based on two key components: the native computer-executable plan, which defines the task to be performed, and a conversational event associated with that plan. A conversational event includes user interactions, such as queries or commands, that trigger the execution of the plan. By analyzing these events, the system identifies patterns and optimizes the interface's ability to interpret and respond to similar inputs in the future. The machine learning process adjusts the interface's algorithms to improve accuracy, efficiency, and user satisfaction. This automated training reduces the need for manual updates and ensures the interface remains effective as user behavior and system capabilities evolve. The approach is particularly useful in dynamic environments where tasks and user interactions vary over time. The result is a more responsive and intelligent conversational interface that adapts to real-world usage.
13. The method of claim 12, further comprising automatically recognizing an exemplary conversational event provided by a user, presenting the exemplary conversational event to a human developer, and prompting the human developer to author a nonnative program implementing the nonnative skill for responding to the exemplary conversational event.
This invention relates to automated systems for developing conversational agents, specifically methods for integrating nonnative skills into such agents. The problem addressed is the difficulty in manually coding responses to diverse conversational events, which requires significant developer effort and expertise. The solution involves a system that automatically detects and recognizes exemplary conversational events provided by users during interactions with a conversational agent. These events are then presented to a human developer, who is prompted to create a nonnative program that implements a new skill for responding to similar events. The nonnative program is designed to handle specific conversational scenarios that fall outside the capabilities of the agent's native programming. This approach streamlines the development process by leveraging user interactions to identify gaps in the agent's functionality and guiding developers to create targeted, reusable responses. The system reduces the need for extensive manual coding by focusing developer effort on high-value, event-specific implementations. This method enhances the agent's adaptability and responsiveness to user needs while minimizing the technical burden on developers.
14. The method of claim 12, wherein the nonnative skill is configured to receive user input from a human user, and to perform a task to respond to the exemplary conversational event based on the user input, the method further comprising automatically recognizing an exemplary conversational event provided by a user, presenting the exemplary conversational event to a human demonstrator, and prompting the human demonstrator to provide user input to the nonnative skill, thereby causing the nonnative skill to perform the task to respond to the exemplary conversational event.
This invention relates to systems for training and utilizing nonnative skills in conversational agents, addressing the challenge of enabling artificial intelligence (AI) systems to handle complex, context-dependent interactions without extensive pre-programmed responses. The technology involves a method for configuring a nonnative skill to receive user input from a human user and perform tasks in response to conversational events. The system automatically recognizes an exemplary conversational event provided by a user, such as a question or command, and presents it to a human demonstrator. The demonstrator is prompted to provide input to the nonnative skill, which then executes a task to respond to the event. This approach allows the AI to learn and adapt to new conversational scenarios dynamically, improving its ability to handle real-world interactions. The method ensures that the nonnative skill can process and respond to user inputs in a way that mimics human-like understanding and interaction, enhancing the overall performance of conversational agents in applications like virtual assistants, customer service bots, and interactive AI systems. The system leverages human input to refine and expand the AI's capabilities, making it more versatile and responsive to diverse user needs.
15. The method of claim 12, further comprising automatically defining the exemplary conversational event based on automatically recognizing input data that is received by the nonnative skill during execution of the nonnative skill.
This invention relates to automated conversational systems, specifically improving the handling of nonnative skills in conversational agents. The problem addressed is the difficulty in dynamically adapting conversational interactions when a user invokes a skill outside the agent's primary domain, leading to disjointed or ineffective responses. The method involves a conversational agent that executes a nonnative skill, which is a function or capability not natively integrated into the agent's core system. During execution of this nonnative skill, the agent automatically recognizes and processes input data received from the user. Based on this input, the agent dynamically defines an exemplary conversational event, which represents a structured representation of the interaction context. This event includes details such as user intent, relevant parameters, and interaction history, allowing the agent to maintain coherence and context awareness even when operating outside its native capabilities. The defined event can then be used to guide subsequent interactions, ensuring smoother transitions and more accurate responses. This approach enhances the agent's ability to handle cross-domain interactions without requiring manual configuration or predefined rules for each possible nonnative skill.
16. The method of claim 15, wherein the nonnative skill is a legacy skill of a legacy automated assistant, and wherein the input data includes an utterance the legacy skill is configured to receive.
This invention relates to automated assistant systems, specifically methods for enabling a nonnative skill to process input data originally intended for a legacy skill. The problem addressed is the inability of newer or nonnative skills to handle input data designed for legacy automated assistant skills, which can lead to inefficiencies and user frustration. The solution involves a method where a nonnative skill is configured to receive and process input data that includes an utterance the legacy skill was originally designed to handle. This allows the nonnative skill to leverage the legacy skill's input data while providing enhanced or alternative functionality. The method may involve translating or adapting the input data to ensure compatibility with the nonnative skill, ensuring seamless integration between different skill versions or types. The approach improves system flexibility and user experience by enabling newer skills to work with existing input data structures without requiring extensive modifications to the legacy system. This is particularly useful in environments where multiple versions of skills coexist or where legacy skills are being phased out. The method ensures continuity and backward compatibility while allowing for innovation in skill development.
17. The method of claim 15, wherein the nonnative skill is a graphical user interface program, and wherein the input data is associated with a graphical user interface input field.
A system and method for training a machine learning model to perform nonnative skills, particularly in the context of graphical user interface (GUI) programs. The invention addresses the challenge of enabling a machine learning model to execute tasks outside its original training scope, such as interacting with GUI elements. The method involves receiving input data associated with a GUI input field, where the input data may include user interactions like clicks, text entries, or selections. The trained model processes this input data to generate an output that corresponds to a desired action within the GUI program, such as filling a form, navigating menus, or executing commands. The system may also include a training phase where the model learns to map input data to appropriate GUI actions using labeled examples or reinforcement learning. The invention ensures that the model can adapt to dynamic GUI environments, handling variations in layout, element types, and user input patterns. This approach enhances automation in software testing, accessibility tools, and user assistance applications by enabling models to perform complex GUI interactions without explicit programming.
19. The method of claim 18, wherein the nonnative skill includes a program configured for execution in an execution environment defined by a virtual machine, and wherein automatically recording computational events associated with calling any other function not among the predefined plurality of named functions includes configuring the virtual machine to automatically record each instruction executed by the virtual machine.
This invention relates to monitoring and recording computational events in a virtual machine environment, particularly for tracking nonnative skills or programs executed within a virtual machine. The problem addressed is the difficulty in capturing detailed execution behavior of programs that are not part of a predefined set of named functions, especially in virtualized environments where traditional monitoring techniques may not apply. The method involves automatically recording computational events associated with calling any function outside a predefined set of named functions. Specifically, when the nonnative skill is a program designed to run in a virtual machine, the system configures the virtual machine to automatically record each instruction executed by the virtual machine. This ensures comprehensive tracking of all execution steps, even for functions not explicitly listed in the predefined set. The approach leverages the virtual machine's execution environment to capture detailed runtime behavior, providing visibility into how nonnative programs interact with the system. This is particularly useful for debugging, security analysis, and performance optimization in virtualized environments where traditional monitoring tools may lack granularity. The solution ensures that all computational events, including those from unanticipated or dynamically generated functions, are logged for further analysis.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
October 4, 2019
December 13, 2022
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.