Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for selecting actions to be performed by an agent interacting with an environment. In one aspect, a system includes a language encoder model that is configured to receive a text string in a particular natural language, and process the text string to generate a text embedding of the text string. The system includes an observation encoder neural network that is configured to receive an observation characterizing a state of the environment, and process the observation to generate an observation embedding of the observation. The system includes a subsystem that is configured to obtain a current text embedding of a current text string and a current observation embedding of a current observation. The subsystem is configured to select an action to be performed by the agent in response to the current observation.
Legal claims defining the scope of protection, as filed with the USPTO.
5. The method of claim 3, wherein the language encoder model is a recurrent neural network.
6. The method of claim 3, wherein the language encoder model is a bag-of-words encoder.
7. The method of claim 3, wherein the current observation embedding is a feature matrix of the current observation, and wherein the current text embedding is a feature vector of the current text string.
9. The method of claim 1, wherein at each of the plurality of time steps, the current text string is a natural language instruction for the agent for performing the current task.
13. The method of claim 1, wherein the current text string is the same for each observation received during the performance of the current task.
14. The method of claim 1, wherein the current text string is different from a preceding text string received during the performance of the current task.
20. The non-transitory computer storage media of claim 18, wherein the language encoder model is a recurrent neural network.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
May 16, 2022
June 11, 2024
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.