World Model Based Distributed Learning for AI Agents

This application claims priority to U.S. Provisional Application No. 63/781,808, filed Apr. 1, 2025, the disclosure of which is herein incorporated by reference in its entirety for all purposes.

The present disclosure generally relates to autonomous vehicles, and more specifically, systems and methods for generating a vehicle path prediction for an autonomous vehicle through lightweight information sharing between autonomous vehicles.

Motion planning for autonomous vehicles presents a multi-agent decision making problem where each agent benefits from knowledge of the trajectory of other agents to improve their own motion planning. One approach to solving the multi-agent decision-making problem is the use of reinforcement learning (RL) where each agent performs distributed learning to make trajectory predictions. However, individual agents performing distributed RL techniques tend to suffer from insufficient information and poor prediction issues due to partial observability and non-stationarity. For example, in an autonomous vehicle environment, each vehicle may only have access to limited sensory data about its immediate surroundings, leading to incomplete knowledge of the state of the environment. As other vehicles change their driving behaviors or traffic conditions evolve, the predictions made by an agent can quickly become outdated or inaccurate, complicating decision-making processes and potentially resulting in suboptimal actions that could affect overall system performance and safety.

In order to tackle these challenges, information sharing is often employed. However, in autonomous vehicle systems, the agents (e.g., vehicles, robots, etc.) interact in a high-dimensional environment that utilizes multiple variables and states to make decisions. Such high-dimensional environment brings prohibitively large communication overhead when sharing information and generating predictions computationally intractable.

Consequently, there is a need for approaches that can effectively handle both the requirements for information sharing and the computational demands of high-dimensional environments.

In some aspects, the techniques described herein relate to a method for generating a trajectory prediction for a vehicle, the method including: obtaining, by one or more processors, first sensory data and first waypoint data associated with a first vehicle; receiving, by the one or more processors, a latent state and a latent intention from a second vehicle, wherein the latent state represents encoded second sensory data associated with the second vehicle, wherein the latent intention represents encoded second waypoint data associated with the second vehicle; fusing, by the one or more processors, the first sensory data and the latent state into a latent fused state; applying, by the one or more processors, the latent fused state, the latent intention, and the first waypoint data into a motion planning module to generate a trajectory prediction for the first vehicle; and applying, by the one or more processors, the trajectory prediction to an autonomous control system to generate control commands for controlling the first vehicle.

In some aspects, the techniques described herein relate to a computer system for generating a trajectory prediction for a vehicle, including: one or more processors, and a tangible, non-transitory memory coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to: obtain first sensory data and first waypoint data associated with a first vehicle; receive a latent state and a latent intention from a second vehicle, wherein the latent state represents encoded second sensory data associated with the second vehicle, wherein the latent intention represents encoded second waypoint data associated with the second vehicle; fuse the first sensory data and the latent state into a latent fused state; apply the latent fused state, the latent intention, and the first waypoint data into a motion planning module to generate a trajectory prediction for the first vehicle; and apply the trajectory prediction to an autonomous control system to generate control commands for controlling the first vehicle.

In some aspects, the techniques described herein relate to a non-transitory computer-readable medium storing executable instructions for generating a trajectory prediction for a vehicle that, when executed by one or more processors, causes the one or more processors to: obtain first sensory data and first waypoint data associated with a first vehicle; receive second sensory data and second waypoint data from a second vehicle; fuse the first sensory data and the second sensory data into a fused sensory data; encode the fused sensory data into a latent fused state, the first waypoint data, and the second waypoint data; apply the latent fused state, the encoded first waypoint data, and the encoded second waypoint data into a motion planning module to generate a trajectory prediction for the first vehicle; and apply the trajectory prediction to an autonomous control system to generate control commands for controlling the first vehicle.

Information sharing among agents (e.g., vehicles) is one technique that may improve the performance in distributed reinforcement learning (RL). More particularly, information sharing mitigates the challenge of insufficient information available to agents for informed decision making. In the multi-agent network, each agent only has incomplete and noisy state information due to their partial observability of the driving environment. Meanwhile, from a single agent's perspective, the single agent's environment would be non-stationary as interacting agents adapt their policies. For example, when a vehicle is driving, the single agent's environment changes, making the information received by the agents to also be non-stationary.

However, conventional information sharing approaches, such as using a central server as a repository for global information or peer-to-peer communication where agents to share information to each other directly through communication channels, may be inefficient in the high-dimensional environments. For instance, the high-dimensional nature of the state (e.g., vehicle sensory data) and action spaces (e.g., waypoints) may lead to prohibitively large communication overhead. Said another way, in these conventional approaches, the data is so large that the act of transmission and reception causes the data to be outdated and/or the data involves so many dimension that traditional approaches to solving multi-agent decision-making problems (such as implement game theory-based approaches) would take too long to process.

Another major bottleneck lies in agents' constrained capability of making predictions of their intertwined environment dynamics. In particular, direct prediction (e.g., vehicle trajectory prediction) in the high-dimensional space would be computationally complex with the exponential increase of the state space dimension in multi-agent settings.

The techniques of the present disclosure relate to using a world model comprising an encoder model, a decoder model, and a prediction model (e.g., motion planning module) to derive generalizations. The generalizations can involve understanding and predicting the behavior of the environment or agents such as predicting vehicle paths, anticipating other agents' behavior, recognizing unseen obstacles, etc.

To overcome the challenge related to high dimensional data, present techniques may implement an encoder model configured to reduce high dimensional sensory data (e.g., camera data, radar data, LiDAR data, GPS data, etc.) and waypoints (e.g., points along a planned trajectory) into low dimensional latent state and latent intention. An agent may then share the low-dimensional latent state and latent intention information to other agents for use in a locally-implement RL process. As a result, by encoding the sensed data into a low-dimensional latent information, the communication overhead that exists in a conventional technique is significantly reduced, as low-dimensional representations are far more efficient to transmit. This approach allows each agent to receive essential, compressed information about other agents' states and intentions without needing to process large volumes of sensory data.

Additionally, each agent may comprise a decoder model that may decode the low-dimensional latent state and the latent intention into high-dimensional sensory data and the waypoints. In some embodiments, each agent may generate a bird's eye view (BEV) using the high-dimensional sensory data. It should be appreciated that the decoded feature space may still have lower dimensionality as compared to the raw sensor data, thereby facilitating more efficient prediction speed. Therefore, the decoded latent state of sensory data may be labeled as environment representation data. By using the encoder model and the decoder model, the present techniques improve flexibility to the multi-agent network, allowing each agent to interpret and utilize the shared latent information effectively. This approach enables agents to reconstruct detailed environmental data and planned paths from other agents without incurring the computational and communication costs associated with direct high-dimensional data sharing. Moreover, as long as the latent information is in a common format, different agents may have different decoders to decode the latent information into a preferred format for the motion planning and prediction modules implemented at that agent. Thus, the instant techniques may facilitate interoperability between agents that utilize different feature spaces for performing motion planning and prediction.

Advantageously, the encoder model and the decoder model may help reduce the communication overload that existed in conventional techniques. In conventional techniques, each additional agent contributes to a significant increase in data transmission load, leading to slower communication and potential delays. By contrast, the encoder model reduces high-dimensional sensory data and waypoints into compact, low-dimensional representations, making it far quicker and more efficient to transmit and receive data among agents. This compression significantly reduces wait times for data transmission and reception, and the reception of latent information from multiple agents contemporaneously, thereby further improving the knowledge of the local environment, enabling faster communication, and minimizing network congestion.

Moreover, the reduced data size also allows for better scalability, as additional agents can join the network without causing exponential increases in communication overhead. Additionally, the encoder and decoder architecture enhances privacy and data security by sharing only essential latent information rather than raw sensory data. Without having a proper decoder to decode the latent information (latent state and latent intention), other malicious agents may not be able to decode the latent information necessary to obtain full sensory data and waypoints.

Additionally, the techniques of the present disclosure may derive generalizations, including vehicle trajectory prediction, other vehicle states, traffic flow patterns, pedestrian behavior, and potential collision risks, by applying the prediction model using the latent information. By operating on low-dimensional latent representations rather than high-dimensional raw data, the present disclosure significantly reduces the computational complexity involved in generating these generalizations. This approach requires less processing power, making it more efficient and enabling faster prediction times. In contrast, conventional techniques relied on processing full sensory data for each generalization, leading to increased computational demands, slower predictions, and more resource-intensive operations. By focusing on latent representations, the present disclosure enables quicker, real-time derivation of essential insights, allowing agents to make informed decisions in complex environments without the delays and computational overhead of traditional methods

Of course, it should be appreciated that the advantages and technical improvements described above and elsewhere herein are not the only advantages and/or technical improvements that may be realized as a result of the techniques described herein. Other advantages and/or technical improvements to the functioning of a computer itself or other technologies or technical fields may be apparent to one of ordinary skill in the art. Moreover, while described herein primarily in the health care claims context, the techniques described herein may be readily applied in any suitable field for any suitable purpose.

1 FIG. 100 108 200 120 122 108 120 200 108 108 illustrates a block diagram of an exemplary autonomous vehicle data systemon which the exemplary methods described herein may be implemented. The high-level architecture includes both hardware and software applications, as well as various data communications channels for communicating data between the various hardware and software components. A vehiclemay include a vehicle computing system, one or more sensors, and a communication component. The vehiclemay obtain information (e.g., sensory data) of its surroundings using the one or more sensors. The vehicle computing systemmay utilize this information to operate the vehicleor to assist the vehicle operator in operating the vehicle.

108 200 108 200 120 108 200 The vehiclemay comprise the vehicle computing system, which may be permanently or removably installed in the vehicle. The vehicle computing systemmay interface with the one or more sensorswithin the vehicle(e.g., a digital camera, a LIDAR sensor, a RADAR sensor, an ultrasonic sensor, an infrared sensor, a millimeter wave sensor, an ignition sensor, an odometer, a system clock, a speedometer, a tachometer, an accelerometer, a gyroscope, a compass, a geolocation unit, radar unit, etc.), which sensors may also be incorporated within or connected to the vehicle computing system.

108 122 108 108 200 130 112 200 108 108 The vehiclemay further include a communication componentto transmit information to and receive information from external sources, including other vehicles such as vehicleB and vehicleC. The vehicle computing systemmay communicate with the networkover link, respectively. The vehicle computing systemmay collect, generate, process, analyze, transmit, receive, and/or act upon data associated with the vehicle(e.g., sensory data, autonomous operation feature settings, or control decisions made by the autonomous operation features) or the vehicle environment (e.g., other vehicles operating near the vehicle).

200 200 108 108 The vehicle computing systemmay be a general-use on-board computer capable of performing many functions relating to vehicle operation or a dedicated computer for autonomous vehicle operation. Further, the vehicle computing systemmay be installed by the manufacturer of the vehicleor as an aftermarket modification or addition to the vehicle.

120 108 120 120 120 108 120 108 108 120 200 The sensorsmay be removably or fixedly installed within the vehicleand may be disposed in various arrangements to provide information to the autonomous operation features. Among the sensorsmay be included one or more of a GPS unit, a radar unit, a LIDAR unit, an ultrasonic sensor, an infrared sensor, an inductance sensor, a camera, an accelerometer, a tachometer, or a speedometer. Some of the sensors(e.g., radar, LIDAR, or camera units) may actively or passively scan the vehicle environment for obstacles (e.g., other vehicles, buildings, pedestrians, etc.), roadways, lane markings, signs, or signals. Other sensors(e.g., GPS, accelerometer, or tachometer units) may provide data for determining the location or movement of the vehicle. Other sensorsmay be directed to the interior or passenger compartment of the vehicle, such as cameras, microphones, pressure sensors, thermometers, or similar sensors to monitor the vehicle operator and/or passengers within the vehicle. Information generated or received by the sensorsmay be communicated to the vehicle computing systemfor use in autonomous vehicle operation.

122 108 108 122 108 In some embodiments, the communication componentmay receive information from external sources, such as other vehiclesB andC. The communication componentmay also send information regarding the vehicleto external sources. The information may include sensory data, waypoint data, and/or hidden state data. The waypoint data may be a series of intermediate points or coordinates that represent a planned path or trajectory for an autonomous vehicle or agent to follow. The hidden state data may represent information about vehicle's past decision (e.g., speed, direction, location, etc.). In some embodiments, the sensory data, waypoint data, and/or hidden state data may be encoded (e.g., latent state and/or latent intention).

122 120 122 108 200 108 To send and receive information, the communication componentmay include a transmitter and a receiver designed to operate according to predetermined specifications, such as the dedicated short-range communication (DSRC) channel, wireless telephony, Wi-Fi, or other existing or later-developed communications protocols. The received information may supplement the data received from the sensorsto implement the autonomous operation features. For example, the communication componentmay receive information that an autonomous vehicle ahead of the vehicleis reducing speed, allowing the vehicle computing systemto adjust in the autonomous operation of the vehicle.

108 108 Information sharing among vehicles, or agents, may help mitigate the challenge of insufficient information that autonomous vehicles often encounter when making informed decisions. For example, the vehicleB may help supplement blind side view of the vehiclewhile driving on the roads.

108 108 108 130 108 108 In some embodiments, the vehicles,B, andC may share information through the networkby sending and receiving latent state and latent intention to reduce the large communication overhead when sending and receiving the information. The latent state may be a low-dimensional latent representation with a small memory footprint of the sensory data, and the latent intention may be low-dimensional latent representation with a small memory footprint of the waypoint data. The vehiclemay obtain the latent state by encoding the sensory data, and may obtain the latent intention by encoding the waypoint data. In some embodiments, the vehiclemay obtain the latent state by encoding both the sensory data and the hidden state data.

120 200 108 200 108 200 108 200 108 108 In addition to receiving information from the sensors, the vehicle computing systemmay directly or indirectly control the operation of the vehicleaccording to various autonomous operation features. The autonomous operation features may include software applications or modules implemented by the vehicle computing systemto generate and implement control commands to control the steering, braking, or throttle of the vehicle. To facilitate such control, the vehicle computing systemmay be communicatively connected to control components of the vehicleby various electrical or electromechanical control components (not shown). When a control command is generated by the vehicle computing system, the command may thus be communicated to the control components of the vehicleto affect a control action. In embodiments involving fully autonomous vehicles, the vehiclemay be operable only through such control components (not shown). In other embodiments, the control components may be disposed within or supplement other vehicle operator control components (not shown), such as steering wheels, accelerator or brake pedals, drive mode switches, or ignition switches.

108 200 In some embodiments, the vehiclemay comprise an autonomous control system that generates control commands based on information (e.g., trajectory prediction) provided by the vehicle computing system. This autonomous control system may process the received information, generate corresponding control commands, and transmit them to the vehicle's operational subsystems (e.g., control components) to execute driving decisions. For example, the control system may regulate steering, braking, or throttle based on the predicted trajectory, ensuring precise vehicle control.

130 130 112 200 130 130 The networkmay be a proprietary network, a secure public internet, a virtual private network or some other type of network, such as dedicated access lines, plain ordinary telephone lines, satellite links, cellular data networks, combinations of these. The networkmay include one or more radio frequency communication links, such as wireless communication linkwith vehicle computing system, respectively. Where the networkcomprises the Internet, data communications may take place over the networkvia an Internet communication protocol.

108 108 108 112 112 130 100 108 108 108 Other vehiclesB andC may perform similarly to the vehicle, and may communicate with each other via linksB andC through the network. Although the systemis shown to include three vehicles,B, andC, it should be understood that different numbers of vehicles may be utilized.

2 FIG.A 200 200 is an expanded block diagram of a vehicle computing system, in accordance with various aspects of the present disclosure. Generally speaking, the vehicle computing systemobtains first sensory data and first waypoint data; receives a latent state and a latent intention from other vehicles; decode the latent state and the latent intention into a environment representation data and a second waypoint data; fuse the first sensory data and the second environment representation data into a fused sensory data; encode the fused sensory data into a latent fused state, and the first waypoint data; apply the latent fused state, the latent intention, and the encoded first waypoint data into a motion planning module to generate a trajectory prediction.

The trajectory prediction may refer to an estimated future trajectory of a vehicle and/or other moving objects within an environment. A trajectory prediction may include a set of predicted future positions and movement patterns derived from latent state data, latent intention data, sensory data, and/or shared information from other agents, as further described below. By representing an expected future state of the vehicle and its surroundings, the trajectory prediction enables proactive decision-making for navigation, collision avoidance, and cooperative maneuvers.

The trajectory prediction may also include a path prediction. The path prediction may refer to a set of predicted future positions that define a planned route for a vehicle within an environment. A path prediction may be generated based on waypoint data, sensory data, latent state data, and/or shared information from other agents to determine an intended sequence of movement. The path prediction may account for road geometry, traffic regulations, and expected maneuvers such as lane changes, turns, or stops. By providing a structured route estimation, the path prediction may enable precise motion planning and facilitate navigation in dynamic environments.

200 202 206 204 206 206 206 1 206 2 206 3 206 206 1 206 2 206 3 206 206 200 200 2 FIG. 1 FIG. The vehicle computing systemincludes one or more processors, one or more memories, and a networking interface. The memoriesincludes world model moduleA comprising an encoder moduleA, a decoder moduleA, and a motion planning moduleA, a perception moduleB comprising a segmentation moduleB, a classification moduleB, and a tracking moduleB, a transmission moduleC, and a bird's-eye view moduleD. In some embodiments, the vehicle computing systemofcan be the vehicle computing systemof.

206 206 The perception moduleB may generate perception signals descriptive of a current state of the autonomous vehicle's environment. It is understood that the term “current” may actually refer to a very short time prior to the generation of any given perception signals, e.g., due to the short processing delay introduced by the perception moduleB and other factors.

206 1 120 206 1 206 1 The segmentation moduleBmay generally be configured to identify distinct objects within the sensory data (e.g., obtained from sensorsB) representing the sensed environment. Depending on the embodiment and/or scenario, the segmentation task may be performed separately for each of a number of different types of sensory data, or may be performed jointly on a fusion of multiple types of sensory data. In some embodiments where lidar devices are used, the segmentation moduleBmay analyze point cloud frames to identify subsets of points within each frame that correspond to probable physical objects in the environment. In other embodiments, the segmentation moduleBmay jointly analyze lidar point cloud frames in conjunction with camera image frames to identify objects in the environment. Other suitable techniques, and/or data from other suitable sensor types, may also be used to identify objects. It is noted that, as used herein, references to different or distinct “objects” may encompass physical things that are entirely disconnected (e.g., with two vehicles being two different “objects”), as well as physical things that are connected or partially connected (e.g., with a vehicle being a first “object” and the vehicle's hitched trailer being a second “object”).

206 1 206 1 206 1 The segmentation moduleBmay use predetermined rules or algorithms to identify objects. For example, the segmentation moduleBmay identify as distinct objects, within a point cloud, any clusters of points that meet certain criteria (e.g., having no more than a certain maximum distance between all points in the cluster, etc.). Alternatively, the segmentation moduleBmay utilize a neural network that has been trained to identify distinct objects within the environment (e.g., using supervised learning with manually generated labels for different objects within test data point clouds, etc.), or another type of machine learning based model.

206 1 206 1 For example, the segmentation moduleBmay process the sensory data through a series of convolutional layers of the neural network. Each convolutional layer may apply filters that detect various features within the input, such as edges, textures, and object boundaries. The segmentation module may tune the filters to highlight relevant details, allowing the neural network to identify key components within the scene, such as other vehicles, lane markers, pedestrians, or obstacles. By focusing on these critical elements, the segmentation moduleBmay begin to condense the sensory input, stripping away unnecessary background details and noise.

206 1 In another example, following the convolutional layers, the segmentation moduleBmay pass the sensory data through pooling layers of the neural network. Pooling layers may select the most prominent feature within a region, ensuring that key signals are preserved even as the overall spatial resolution is reduced. Pooling layers may help to capture larger patterns in the environment while discarding finer details that may be less relevant for decision-making.

206 2 206 1 206 1 206 2 206 2 206 2 The classification moduleBmay generally be configured to determine classes (labels, categories, etc.) for different objects that have been identified by the segmentation moduleB. Like the segmentation moduleB, the classification moduleBmay perform classification separately for different sets of the sensory data, or may classify objects based on data from multiple sensors, etc. Moreover, and also similar to the segmentation moduleB, the classification moduleBmay execute predetermined rules or algorithms to classify objects, or may utilize a neural network or other machine learning based model to classify objects.

206 2 206 1 206 2 In some implementations, the classification moduleBmay communicate with an object database (not depicted) that stores information associated with object types. For example, the object database may include information that indicates how an object of the corresponding object type should appear in a point cloud. In some implementations, this indication may be a generic model for an object of the particular type. As one example, an object database record for a particular model of car may include a three-dimensional model of the car, to which objects identified by the segmentation moduleBare compared during the classification process. In some additional implementations, the model includes indications of particular features of the object that have known shapes (e.g., a license plate, a tire, manufacturer emblem, etc.). As will be described below, this model of the object, including the particular features thereof, may be used to detect whether objects identified by the segmentation moduleBactually have a skewed shape, or instead have a distorted appearance in the point cloud frame due to rolling shutter distortion.

206 3 206 1 206 2 206 1 206 3 206 1 206 2 206 3 206 1 206 2 206 3 The tracking moduleBmay generally be configured to track distinct objects over time (e.g., across multiple lidar point cloud or camera image frames). The tracked objects are generally objects that have been identified by the segmentation moduleB, but may or may not be objects that were classified by the classification moduleB, depending on the embodiment and/or scenario. The segmentation moduleBmay assign identifiers to identified objects, and the tracking moduleBmay associate existing identifiers with specific objects where appropriate (e.g., for lidar data, by associating the same identifier with different clusters of points, at different locations, in successive point cloud frames). Like the segmentation moduleBand the classification moduleB, the tracking moduleBmay perform separate object tracking based on different sets of the sensory data, or may track objects based on data from multiple sensors. Moreover, and also similar to the segmentation moduleBand the classification moduleB, the tracking moduleBmay execute predetermined rules or algorithms to track objects, or may utilize a neural network or other machine learning model to track objects.

206 1 206 1 206 1 206 206 1 200 200 The encoder moduleAmay encode high dimensional data (e.g., bigger feature space) into low dimensional data (e.g., smaller feature space). For example, the encoder moduleAmay encode high dimensional sensory input (e.g., images from cameras) into a low-dimension latent state. In another example, the encoder moduleAmay encode high dimensional perception signals from the perception moduleB into low dimensional latent state. The encoder moduleAmay pass the sensory input through several layers of a neural network (e.g., recurrent neural network (RNN)). The neural network may then condense the relevant features into a compact, low-dimensional representation or latent state, which captures essential information while discarding irrelevant details. The vehicle computing systemmay use the latent state for decision-making or share with other agents (e.g., vehicles). By encoding sensory input into latent states, the vehicle computing systemmay minimize memory and computational requirements, enabling efficient communication and improving prediction capabilities across the agent network. Latent representations may have a much smaller memory footprint compared to the high-dimension sensory inputs, making lightweight information sharing much more feasible in practice.

206 1 206 1 The encoding moduleAmay apply the high-dimensional sensory data into a set of fully connected (dense) layers of the neural network. These layers may take the two-dimensional feature maps (e.g., sensory data such as images) and transform them into a compact, fixed-size vector known as the latent state. The fully connected layers may ‘flatten’ the features, combining the features into a concise representation that encapsulates the core information about the environment. This transformation may serve as a bottleneck, forcing the encoder moduleAto retain only the most critical aspects of the input, ensuring that the resulting latent state is both compact and highly informative.

206 1 206 1 In some embodiments, the encoder moduleAmay processes the high-dimensional sensory data and the high-dimensional hidden state data through a series of recurrent layers of the neural network. These recurrent layers, such as Long Short-Term Memory (LSTM) units or Gated Recurrent Units (GRUs), are designed to capture temporal dependencies and relationships across sequential inputs. By iteratively updating the hidden state, the recurrent layers encode both the current sensory data and the historical context contained in the hidden state, allowing the encoder moduleAto generate a compact, low-dimensional latent state representation. This representation combines the most relevant information from past and present inputs, enabling downstream processes to make informed predictions or decisions based on the condensed and temporally aware encoding of the environment.

206 1 108 The encoder moduleAmay apply the same neural network to the waypoint data of the vehicleto determine a latent intention. The waypoint data may be a sequence of target coordinates or positions that outline the vehicle's planned trajectory over a given time horizon. By encoding the waypoint data into the latent intention, the encoder provides a compact representation of the vehicle's intended path, which can be shared with other agents.

200 206 3 The latent data including the latent state data and the latent intentions may be deemed as “locally sufficient information” for the vehicle computing systemto make decisions and improve predictions using the motion planning moduleA.

206 2 206 1 206 2 200 206 2 The decoder moduleAmay decode the latent data encoded by the encoder moduleA. The latent data may comprise latent state (e.g., encoded sensory data) and/or latent intention (e.g., encoded waypoint data). The decoder moduleAmay decode the lower-dimensional latent data to a higher-dimension sensory data and/or the waypoint data. The vehicle computing system, upon receiving the latent data, may use the decoder moduleAto decode one or more latent state and the latent intention, and obtain one or more sensory data and waypoint data from other vehicles.

206 2 206 2 206 1 a The decoder moduleAmay decode the latent data by passing the latent data through several layers of a neural network (e.g., Recurrent Neural Network). These layers may transform the compact, low-dimensional latent representation back into a higher-dimensional form that resembles the original sensory data or waypoint data. For example, the decoder modulemay use deconvolutional layers to up-sample the latent data, gradually reconstructing details that were previously condensed by the encoder moduleA. Through this process, the decoder may recover key spatial and temporal details necessary for accurate perception and navigation.

206 2 200 206 2 200 The decoder moduleAmay train the neural network layers within the decoder to reconstruct high-fidelity representations of sensory inputs, such as images, enabling the vehicle computing systemto interpret detailed information about nearby agents' positions, velocities, and intentions. For example, the decoder moduleAmay regenerate an approximate image or positional data that provides insights into the environment surrounding other vehicles. This capability allows the vehicle computing systemto assess the relative positions of objects, obstacles, and road features, even when the original high-dimensional data was not directly shared.

206 2 206 2 200 Additionally, the decoder moduleAmay apply recurrent layers to the latent intention data, which may capture the sequential nature of waypoint paths and predicted trajectories. These recurrent layers enable the decoder moduleAto reconstruct temporal sequences, such as the planned trajectory and expected maneuvers of other vehicles. By decoding latent intentions in this way, the vehicle computing systemmay gain foresight into other vehicles' future actions, allowing for more informed planning and coordination in multi-agent environments.

200 206 2 After decoding, the vehicle computing systemmay utilize the reconstructed sensory and waypoint data to enhance its situational awareness and decision-making processes. By integrating this data with its own sensory inputs, the decoder moduleAmay achieve a more comprehensive view of the environment.

206 4 206 4 206 2 206 3 206 4 206 4 206 4 206 4 200 206 4 The prediction componentAmay process the latent state to generate prediction signals descriptive of one or more predicted future states of the autonomous vehicle's environment. For a given object, for example, the prediction componentAmay analyze the type/class of the object (as determined by the classification moduleB) along with the recent tracked movement of the object (as determined by the tracking moduleB) to predict one or more future positions of the object. As a relatively simple example, the prediction componentAmay assume that any moving objects will continue to travel in their current direction and with their current speed, possibly taking into account first- or higher order derivatives to better track objects that have continuously changing directions, objects that are accelerating, and so on. In some embodiments, the prediction componentAmay also predict movement of objects based on more complex behaviors. For example, the prediction componentAmay assume that an object that has been classified as another vehicle will follow rules of the road (e.g., stop when approaching a red light), and will react in a certain way to other dynamic objects (e.g., attempt to maintain some safe distance from other vehicles). The prediction componentAmay inherently account for such behaviors by utilizing a neural network or other machine learning model, for example. It should be appreciated that if an object corresponds to an agent from which the vehicle computing systemreceived encoded latent information, the latent information (either encoded or decoded) may be analyzed by the prediction componentAto predict futures states of the agent.

206 5 200 180 The motion plannerAmay process the latent state and/or the prediction signals to generate waypoint data, or a series of intermediate points or coordinates that represent a planned path or trajectory. The vehicle computing systemmay utilize its own latent state and prediction signals (vehicle), such as tracked objects and environmental cues, to create waypoints for navigating the vehicle.

206 3 206 3 206 4 206 5 206 5 206 3 206 3 The routing planning moduleAmay also utilize latent state and latent intention (e.g., encoded waypoint data) from other vehicles to derive generalizations (e.g., predicted future states of the environment, trajectory predictions, etc.) along with its own latent state and latent intention. For example, the routing planning moduleAmay use the prediction componentAto predict future states of the environment, and use the motion plannerAto generate a trajectory prediction. By leveraging the shared latent states and latent intentions from nearby agents, the motion plannerAcan derive generalizations about the collective environment. For example, it may anticipate lane changes, braking behaviors, or traffic flow patterns based on other vehicles' intentions and trajectories. Combining its own latent intention, latent state, sensory data, and shared information enables the routing planning moduleAmay determine well-informed trajectory predictions and plan cooperative maneuvers. The motion planning moduleAmay control the vehicle based on the predicted path and planned cooperative maneuvers, such as yielding, merging, or adjusting speeds in response to anticipated behaviors of nearby agents.

206 3 200 206 3 206 3 t+k t+k 3 FIG.B The motion planning moduleAmay also utilize a memory-augmented neural network (e.g., a recurrent neural network) to process the latent data (e.g., latent state and latent intention) and derive generalizations. These generalizations enable the vehicle computing systemto predict future states (e.g., future latent state {circumflex over (z)}) and plan its actions (e.g., action a) accordingly. Specifically, the motion planning moduleAmay determine insights such as vehicle trajectory predictions, anticipated behaviors of nearby agents, traffic flow patterns, potential collision risks, and optimal navigational strategies. By leveraging these capabilities, the motion planning moduleAenables the vehicle to plan and control safe and efficient maneuvers based on the trajectory prediction, such as adjusting speed, changing lanes, or preparing to yield, thereby improving its adaptability and responsiveness to dynamic, real-world driving environments. Details of deriving generalizations can be found in.

206 206 200 108 200 200 108 200 The transmission moduleC may choose to send and/or receive information from vehicles that are within a predefined distance threshold. For example, the vehicle may choose to send and receive the latent state and/or latent intention among vehicles within 5-mile radius. The predefined distance threshold may be based on a broadcasting signal range of the first vehicle, or may be based on a global positioning system (GPS) distance. In some embodiments, the transmission moduleC may prioritize certain vehicles in sending the latent state data and/or latent intention data. For example, the vehicle computing systemmay prioritize sending the latent state and/or intention data to vehicles in close proximity or those likely to be affected by the planned maneuvers of the vehicle, such as those in adjacent lanes or directly behind when preparing to change lanes. Additionally, if the vehicle computing systemdetects that certain vehicles are approaching quickly or are in potential collision paths, the vehicle computing systemmay transmit updated latent information more frequently to enable better situational awareness for those vehicles. Conversely, for vehicles farther away or not directly relevant to the vehicle's immediate path, the vehicle computing systemmay limit or omit latent information transmission to conserve bandwidth and processing resources.”

206 200 206 108 108 108 200 108 In some embodiments, the transmission moduleC may choose which latent information (or subset of the sensory data) to send out to other vehicles. When the vehicle computing systemchooses to send out latent state and/or the latent intention (or sensory data and/or waypoint data) to other vehicles, the transmission moduleC may determine which vehicles may need what information. For example, in a lane-changing task, it may suffice for the vehicleto send only its lateral movement intentions and speed data to vehicles (e.g., vehicleB and vehicleC) in adjacent lanes, as these vehicles would be most impacted by an upcoming lane shift. Specifically, the vehicle computing systemmay transmit the vehicle's intended trajectory within the lane, projected speed changes, and any planned accelerations or decelerations, which could help nearby vehicles anticipate and react appropriately to the lane change.

206 108 108 108 108 108 The transmission moduleC may additionally choose which latent information (or subset of the sensory data) to send out to other vehicles based on an absolute position of the vehicleto another vehicle (e.g., vehicleB). For example, the vehiclemay transmit more comprehensive sensory data, such as its immediate surrounding view (e.g., obstacles, lane markings, and nearby vehicles), precise projected acceleration and deceleration, and detailed trajectory predictions, to vehicles within a close range, as these vehicles are more likely to be impacted by the vehicle's maneuvers. In contrast, the vehiclemay transmit less comprehensive sensory data, such as basic information about its current speed, heading, and general movement intentions, to vehicles that are farther away.

206 206 The transmission moduleC may determine which information to send and/or receive using a context-aware filtering mechanism. The context-aware filtering mechanism may analyze the vehicle's current task (e.g., lane-changing, merging, or maintaining speed) and prioritize data that is most relevant to that task. For instance, when performing a merging maneuver, the transmission moduleC may prioritize data on the lateral and longitudinal positions, speeds, and intentions of vehicles in adjacent lanes or those directly ahead and behind in the merging lane. It may deprioritize or ignore data from vehicles several lanes away or those whose movements are unlikely to impact the merging process.

206 206 206 In some embodiments, the transmission moduleC may determine which information to send and/or receive using a prediction-accuracy-driven information sharing mechanism. This mechanism may evaluate prediction errors by comparing previous trajectory predictions including previously predicted latent states and intentions against actual observations over recent time steps. When the prediction error exceeds a predefined threshold, the transmission moduleC may adapt its communication range and exchange further relevant latent information. For instance, when performing a merging maneuver and the prediction error is high, the transmission moduleC may increase the communication range (e.g., by increasing the range of its transceivers) to obtain additional information about vehicles in adjacent lanes or those directly ahead and behind in the merging lane. This adaptive approach ensures only lightweight and targeted information sharing takes place, reducing unnecessary communication overhead while maintaining prediction accuracy.

This context-aware filtering mechanism could be implemented using different approaches. A rule-based system with predefined rules may prioritize or ignore data based on specific tasks and conditions, such as only receiving data from nearby vehicles during merging, which is simple, low-cost, and predictable but less flexible in complex scenarios. Alternatively, a machine learning model could dynamically filter data by recognizing patterns relevant to various situations, such as using reinforcement learning to optimize safe and efficient driving across traffic scenarios, though this requires more computational resources and training data. A hybrid approach may combine rule-based logic for straightforward cases with a machine learning model for complex situations, such as dense traffic merging, balancing simplicity and flexibility for efficient adaptation to varied driving conditions.

206 In some embodiments, the transmission moduleC may use real-time situational analysis, considering environmental factors such as the vehicle's speed, proximity to other vehicles, and the density of traffic, to further refine which information is necessary. Similar to the context-aware filtering mechanism, the situational analysis may involve machine learning algorithms or rule-based systems to assist in making these decisions, allowing the vehicle to dynamically adjust its data-sharing preferences based on changes in traffic or road conditions.

206 200 The transmission moduleC, by focusing on sending and receiving only the most relevant information for the current task (e.g., context aware filtering) and/or the current environment (e.g., real-time situational analysis), may conserve bandwidth, optimize processing resources, and ensure that the vehicle computing systemremains focused on critical data needed for safe and efficient navigation.

206 180 206 206 206 200 260 2 206 The BEV generator moduleD may generate a bird's eye view (BEV) of the vehicle's environment using the sensory data and waypoint data. The BEV generator moduleD may transform raw sensory inputs, such as camera images, LiDAR point clouds, radar data, and GPS coordinates, into a top-down representation of the surroundings. The sensory data may provide real-time information about the vehicle's environment, including the location of other vehicles, lane markings, obstacles, and road features. The waypoint data, which represents the planned trajectory of the vehicle, is overlaid onto the BEV to provide a visual representation of the intended path. By integrating these inputs, the BEV generator module creates a comprehensive and intuitive map that highlights both the current state of the environment and the vehicle's planned movement. In some embodiments, the BEV generator moduleD may combine different BEVs of different vehicles to its own BEV. For example, the BEV generator moduleD may determine a fused BEV based on the fused sensory data and the fused waypoint data. In some embodiments, where the decoded feature space represents a BEV of the environment proximate to the vehicle computing system, the decoder moduleAmay invoke the BEV generatorD when decoding received encoded latent information.

2 FIG.B 1 FIG. 200 180 210 210 210 120 120 illustrates an example software architecture of a vehicle computing systemgenerating waypoints for a vehicle (e.g., vehicle) using sensory data. The software architecture may receive as input M sets of sensory datagenerated by M different sensors, with M being any suitable integer equal to or greater than one. The sensor datamay be data generated by the sensorsof. For example, the sensorsmay include one or more of a GPS unit, a radar unit, a LIDAR unit, an ultrasonic sensor, an infrared sensor, an inductance sensor, a camera, an accelerometer, a tachometer, or a speedometer.

206 206 1 206 2 206 3 206 210 212 206 1 210 206 1 206 2 206 1 206 2 206 3 206 1 206 4 2 FIG.A 2 FIG.A 2 FIG.A The perception moduleB may comprise the segmentation moduleB, the classification moduleB, and the tracking moduleB. The perception moduleB may process the sensory datainto perception signalsdescribing a current state of the autonomous vehicle's environment. The segmentation moduleBmay identify distinct objects within the sensory datausing predetermined rules or algorithms (as specified in segmentation moduleBof). The classification moduleBmay classify (labels, categories, etc.) for different objects that have been identified by the segmentation moduleB(as specified in classification moduleBof). The tracking moduleBmay track distinct objects identified in the segmentation moduleB, and may assign identifiers to the identified objects (as specified in tracking moduleBof).

206 1 212 212 212 206 4 222 206 4 2 FIG.B The encoder moduleAmay then encode the perception signalsand other sensory data that were not processed into perception signals(e.g., GPS data) into a latent state. In some embodiments, the software architecture may use hidden state (e.g., historical waypoint data and historical sensory data) of the vehicle in addition to the sensory data, and encode the hidden state along with the perception signalsand the sensory data to determine the latent state. The prediction componentAmay process the latent state to generate prediction signalsdescriptive of one or more predicted future states of the autonomous vehicle's environment (as specified in prediction componentAof).

220 218 218 218 A mapping componentobtains map data (e.g., a digital map including the area currently being traversed by the autonomous vehicle) and/or navigation data (e.g., data indicating a trajectory for the autonomous vehicle to reach the destination, such as turn-by-turn instructions), and outputs the data (possibly in a converted format) as mapping and navigation signals. In some embodiments, the mapping and navigation signalsmay include other map- or location related information, such as speed limits, traffic indicators, and so on. The signalsmay be obtained from a remote server (e.g., via a cellular or other communication network of the autonomous vehicle, or of a smartphone coupled to the autonomous vehicle, etc.), and/or may be locally stored in a persistent memory of the autonomous vehicle.

206 5 214 222 218 224 200 210 180 206 5 208 2 FIG.B The motion plannerAmay then process the latent state, the prediction signals, and the navigation signalsto generate the waypointand/or a trajectory prediction. The vehicle computing systemmay utilize the software architecture ofand its own sensory datato create waypoints and/or the trajectory prediction for navigating the vehicle (e.g., vehicle). The motion plannerAmay then transmit the trajectory prediction to a control systemto generate control commands for controlling the vehicle.

208 208 208 200 The control systemmay process the trajectory prediction and generate control commands for the vehicle's operational subsystems (e.g., control components), enabling the vehicle to autonomously execute driving decisions. For example, the control systemmay generate and implement commands to regulate the vehicle's steering, braking, or throttle based on the predicted trajectory. In some embodiments, the control systemmay function independently from the vehicle computing system, autonomously generating and executing control signals based on the trajectory prediction to operate the vehicle.

3 FIG.A 2 FIG.B 302 302 302 120 302 120 illustrates a flow diagram of a multi-agent system deriving generalizations (e.g., trajectory predictions). The multi-agent system may comprise a first agentA (e.g., first vehicle) and a second agent (e.g., second vehicle)B. The first agentA may collect a first sensory data (e.g., local observations) from one or more sensors (e.g., one or more sensors) and first waypoint data (e.g., from software architecture of). The second agentB may collect a second sensory data (e.g. local observations) from one or more sensors (e.g., one or more sensorsB) and second waypoint data. The waypoint data may be a sequence of intermediate target points or coordinates guiding the vehicle's path.

302 304 206 1 308 306 302 302 302 308 302 306 308 302 The first agentA may use a first encoderA (e.g., encoder moduleA) to encode the first sensory data into a first latent stateA, and encode the first waypoint data into a first latent intentionA. In some embodiments, the first agentA may define a subset of the first sensory data based on an absolute position of the first agentA to the second agentB, and encode the subset of the first sensory data as the first latent stateA. The first agentA may then transmit the first latent data comprising the first latent intentionA and the first latent stateA to the second agentB.

302 304 308 306 302 302 302 308 302 306 308 302 302 302 The second agentB may use a second encoderB to encode the second sensory data into a second latent stateB, and encode the second waypoint data into a second latent intentionB. In some embodiments, the second agentB may define a subset of the second sensory data based on an absolute position of the second agentB to the first agentA, and encode the subset of the second sensory data as the second latent stateB. The second agentB may then transmit the second latent data comprising the second latent intentionB and the second latent stateB to the first agentA. In some embodiments, the second agentB may select which latent data to send to the second agentB.

302 308 306 308 306 310 206 2 302 302 302 302 302 The first agentA, upon receiving the second latent stateB and the second latent intentionB, may decode the second latent stateB and the second latent intentionB using the decoderA (e.g., decoder moduleA) to determine the second environment representation data and the second waypoint data. The first agentA may then combine the first sensory data and the second environment representation data to generate fused sensory data. The fused sensory data, the first waypoint data, and the second waypoint data may provide the first agentA with an enhanced understanding of its surrounding environment by integrating information from both its own observations and those received from the second agentB. By combining sensory data, the first agentA can build a more complete and accurate representation of nearby vehicles, obstacles, and road conditions, which can improve its situational knowledge when performing trajectory predictions. Similarly, additional waypoint data offers insights into the planned paths of both agents, enabling the first agentA to anticipate potential interactions or conflicts along its trajectory.

302 308 306 308 306 310 302 The second agentB, upon receiving the first latent stateA and the first latent intentionA, may decode the first latent stateA and the first latent intentionA using the decoderB to determine the first environment representation data and the first waypoint data. The second agentB may then combine the first environment representation data and the second sensory data to determine the fused sensory data to determine the fused sensory data.

302 306 306 302 306 306 312 302 302 306 306 314 3 FIG.B 5 FIG. The first agentA may then encode the fused sensory data into a latent fused state, the first waypoint data into the first latent intentionA, and the second waypoint data into the second latent intentionB. The first agentA may then use the latent fused state, the first latent intentionA, and the second latent intentionB to derive a first set of generalizationsA for a plurality of future time intervals. These generalizations may include predictions about the positions, speeds, and trajectories of other agents, as well as anticipated changes in the environment, such as obstacles or road conditions. By simulating potential scenarios over a range of future time steps, the first agentA can assess the likely outcomes of various decisions, enabling it to choose actions that optimize its safety and efficiency. Similarly, the second agentB may then use the fused sensory data and the fused waypoint data to determine the latent fused state, the first latent intentionA, and the second latent intentionB, and derive a second set of generalizationsB for a plurality of future time intervals. Details of forming the generalizations can be found inand.

302 308 308 304 310 304 310 302 308 302 308 In some embodiments, the first agentA, upon receiving the second latent stateB, may combine the first sensory data and the second latent stateB to determine a latent fused state directly. The first encoderA and the first decoderA, being the same encoder/decoder type as the second encoderB and the second decoderB, allow the first agentA to process and integrate the received latent state without requiring explicit decoding. Because both agents utilize the same encoding architecture, the latent stateB may already exist in a structured, compressed form that retains meaningful features relevant to decision-making. Similarly, the second agentA, may combine the second sensory data and the first latent stateA to determine the latent fused state.

206 The multi-agent system using a world model (e.g., world model moduleA) to derive generalizations may be represented in a mathematical framework, with variables indicating different states, spaces, actions, and observations within the environment. This framework may enable each agent (e.g., vehicle) to encode, process, and predict relevant information about its surroundings and the behavior of other agents. Mathematical constructs, such as state transition functions and reward functions, allow agents to evaluate the impact of their actions and learn policies that maximize expected rewards over time. This structured mathematical framework may provide a foundation for encoding complex multi-agent interactions and enables the system to predict future states, determine optimal actions, and generalize across various scenarios.

i i∈N i i∈N i i∈N i i i i ds t t+1 The distributed decision making problem in the multi-agent system may cast as a (partial-observable) stochastic game (S, {A}, P, {r}, {Ω}, γ), where N is the set of N agents in the system, S⊆Ris the state space of the environment, and Ωand Aare the observation space and action space for agent i∈N, respectively. The stochastic game may be a mathematical framework where multiple agents make decisions in an environment (e.g., driving) that change probabilistically. Each agent has a partial observation space Ω, as they can only observe part of the environment, leading to a partially observable setting. Each agent i has its own set of possible actions, represented by A. S may represent all possible states of the environment. P may represent a state transition function going from one state sat time t to another state sat time t+1. Meanwhile, the multi-agent system may assume the state space to be compact, implying that the environment's possible states are limited to a finite region, and assume the action space to be finite, meaning each agent has a limited, countable set of actions. γ∈[0, 1) may represent a discounting factor. The discount factor γ∈[0,1) may determine how much future rewards are valued compared to immediate rewards, where a value closer to 1 places more importance on long-term rewards. In a single agent's perspective, each agent's decision-making problem may be viewed as a Partial-Observable Markov Decision Process (POMDP), as each agent has only partial observations.

i,t i t 1,t N,t t 1,t N,t i i i,t t t+1 t+1 t t i,t i i,t i t t At each time step t, each agent i may choose an action aby following policy π: S→A, and denote the joint action by a=[a. . . , a]∈A, where the joint action aincludes the actions of all agents [a. . . a], and represented by A:=ΠA, which may represent the product of all agents' action spaces. The action amay also be labeled as a waypoint data of agent i at time t. Then the environment may evolve from sto sfollowing the state transition function P(s|s, a) in the domain of S×A×S→[0, 1]. Each agent i may have a partial observation, e.g., sensory inputs of an autonomous vehicle, o∈Ωand receive the reward r:=r(s, a).

Each agent in the multi-agent system may aim to train a world model WM to represent the latent dynamics of the environment and predict the reward r and future latent state z. Each component of the world model may be implemented as a neural network and φ may be the combined parameter vector. The world model WM may comprise following models:

φ φ t 206 3 An augmented model pand the sequence model fmay be part of the memory augmented neural network of the motion planning moduleA. Tmay represent the shared information from other agents at time t. This information may be shared based on a prediction-accuracy-driven approach, where prediction errors are continuously monitored to determine when information sharing is necessary. The prediction error at step k may be characterized by the difference between the predicted state and the true state, which can be improved through effective information sharing as described in the theoretical analysis. In some embodiments, the dynamics predictor may be labeled as state representation predictor.

i,t i,t φ d The world model WM may first learn a latent state z∈Z⊆Rbased on the agent's partial observation from sensory inputs othrough encoding. Moreover, the world model WM may use a recurrent state-space (e.g., RSS) model as an encoder qto capture the context information of the current observation in the latent space by incorporating the hidden state h that provides context and past history data about the vehicle, i.e.,

i,t i,t i,t i,t i,t i,t i,t i,t The encoder processes two key inputs: the hidden state h, which encapsulates historical context about past states and actions, and the observation o, which consists of real-time sensory data (e.g., camera images, LiDAR scans). By combining these inputs, the encoder captures the temporal dependencies and spatial features of the environment, producing a latent state zthat represents a concise yet informative summary of the agent's state. In some embodiments, the concatenation of hand zas the model may be represented as x:=[h, z]∈X.

φ i,t+1 Then a sequence model f, e.g., RNN, may predict the next hidden state hgiven joint action at, i.e.,

i,t+1 i,t i,t φ i,t+1 the sequence model predicts the next hidden state hbased on the current state xand the current action a. fmay be a parameterized function, often implemented as a recurrent neural network (RNN) or similar architecture, designed to capture temporal dependencies and dynamics within sequential data. The next hidden state hacts as a compact representation of the agent's historical context, integrating past observations, actions, and transitions to inform future decisions. This enables the sequence model to handle complex, dynamic environments by retaining relevant temporal information and allowing the vehicle computing system to adapt its predictions and actions as the environment evolves.

i,t+1 i,t+1 φ Based on the determined next hidden state h, the agent may predict next latent state żusing an augmented model p(e.g., memory augmented neural network), i.e.,

i,t+1 i,t+1 The dynamics predictor is a component designed to predict future latent states based on the current hidden state. The dynamics predictor models how the environment is expected to evolve over time. Using the next hidden state h, which encapsulates historical context and temporal dependencies, the dynamics predictor estimates the next latent state ż. This future latent state provides a compact and abstract representation of the anticipated environment, enabling the system to predict key aspects such as the movement of nearby vehicles, changes in traffic patterns, or evolving road conditions.

φ Based on the next model state, WM also predicts the reward using the augmented model p, i.e.,

i,t+1 i,t+1 i,t+1 i,t i,t The reward predictor estimates the reward at the next time step t+1 based on the predicted state x=[h, {circumflex over (z)}]. This reward prediction enables the system to evaluate the potential benefits or consequences of actions taken. Determining a reward rmay use the same augmented model using the hidden state and the latent state z.

t 1 safe 2 comfort 3 time 4 velocity 5 ori 6 target In some embodiments, the reward may be weight sum of five different attributes including safety, comfort, driving time, velocity, and distance to the waypoint. For example, the reward may be represented as R=wR+wR+wR+wR+wR+wR

φ Based on the model state, WM also predicts the continuation using the augmented model pi.e.,

i,t+1 i,t i,t The continue predictor predicts whether the episode or task should continue based on the predicted state x. This flag provides a binary or probabilistic indication of whether the current trajectory or strategy remains viable. Determining a continuation cmay use the same augmented model using the hidden state and the latent state z.

i t i,t i i t i,t i i,t i t i i,t a i ~π i i,t i,t i i,t i,t a i ~π i t i,t i,t i,t π i π t In multi-agent system, each agent i may learn a policy π, i∈N, in a distributed manner, aided by lightweight information sharing among agents. Specifically, the multi-agent system may consider the practical setting where each agent is myopic and treats other agents as part of the environment. The global information (e.g., latent state z(t) and latent intention w(t) from other agents) at time step t may be represented as I, which contains all the information in the system. During the interaction with the environment, agent i may choose an action a~πbased on received information T(I) and current state x. Then the goal of agent i is to find a policy (π(⋅|x,T(I)) that maximizes its value function v(x) Δ=E[Q(x,a)], with Q-function Q(x, a)=E[Σγr] being the expected return when the action ais chosen at state x.

i i,t+1 i,t+1 i 200 The value function v(x) may represent the expected long-term reward an agent i can achieve from a given state xby following a specific policy π. The value function integrates insights from the dynamics predictor (or latent state z), reward predictor, and continue predictor, providing a comprehensive measure of the quality of a state or action. This helps the vehicle computing systemselect actions that maximize the expected cumulative reward while accounting for future states and their potential outcomes.

t+k t+k t+k t+k i i i,t+k t+1 t+1 t+1 t+1 Actions are determined by integrating predictions and evaluations from all components of the generalization module. At each time step k, the system evaluates the predicted future latent states {circumflex over (z)}, the corresponding rewards {circumflex over (r)}, and the likelihood of success ĉ. Using this information, the vehicle computes the value of potential actions âby optimizing its policy πto maximize the value function v(x). Therefore, for time t+1, the action âcan be predicted using the next latent states {circumflex over (z)}, the corresponding rewards {circumflex over (r)}, and the likelihood of success ĉ. For example, the system may select an action that ensures a smooth lane change, reduces collision risk, or optimizes traffic flow. By balancing immediate rewards with long-term goals, the system ensures that its decisions are both reactive to current conditions and proactive in navigating complex environments.

3 FIG.B i i i i i 1,t+k 1,t+k 2,t+k 2,t+k 320 320 320 320 illustrates a state diagram of an interplay between two agents (vehicles) employing a world model to come up with generalizations. The two agents may share latent state and latent intentions [z(t), w(t)] via lightweight communications. In some embodiments, the two agents may share latent state, hidden state, and latent intention [z(t), h(t), w(t)] via lightweight communications. A first agentA, based on the latent state and the latent intention of a second agentB and its own, may derive generalizations including a predicted latent state {circumflex over (z)}and a future action state a, where k represents different time intervals. The second agentB, based on the latent state and the latent intention of the first agentA, may derive generalization including a predicted latent state {circumflex over (z)}and a future action state a.

320 320 206 1 320 320 320 320 320 320 1,t 1,t 1,t 1,t 1,t 1,t 1,t 1,t 1,t 1,t 2 FIG.B The first agentA may collect first sensory data, or first sensory input ousing one or more sensors. The first agentA may then encode (e.g., using the encoding moduleAor the encoder model) the first sensory input oto determine a first latent state z. In some embodiments, the first agentA may encode the first hidden state data hor representation of information from past observations and actions maintained over time, along with the first sensory input to determine a first latent state data z. For example, the hidden state data may be recent positions and speeds of the vehicle. The first agentA may additionally encode the first waypoint data (e.g., derived using the software architecture of) to determine a first latent intention w. The first agentA may then transmit the first latent state zand the first latent intention wto the second agentB. In some embodiments, The first agentA may then transmit the first latent state zand the first latent intention wto the second agentB based on the prediction-accuracy-driven information sharing mechanism.

320 320 320 320 320 320 320 2,t 2,t 2,t 2,t 2,t 1,t 1,t 1,t 2,t 2,t 2,t 2,t 2,t The second agentB may collect second sensory data, or second sensory input ousing one or more sensors. The second agentB may then encode the second sensory input oto determine a second latent state data z. In some embodiments, a second hidden state data h, or representation of information from past observations and actions maintained over time to provide context for decision-making, may be used along with the sensory input to determine a second latent state data z. The combination of the first hidden state data hand the first latent state zmay be represented as a second state x. The second agentB may additionally encode second waypoint data to determine a second latent intention w. The second agentB may then transmit the second latent state zand the second latent intention wto the first agentA. In some embodiments, the second agentB may then transmit the second latent state zand the second latent intention wto the first agentA based on the prediction-accuracy-driven information sharing mechanism.

320 320 320 2,t 2,t 1,t 1 In some embodiments, the first agentA may decode the second latent state zand the second latent intention winto the second environment representation data and the second waypoint data. The first agentA may then combine the first sensory data and the second environment representation data to generate first fused sensory data. The first agentA may then encode the first fused sensory data, the first waypoint data, and the second waypoint data to obtain a first latent fused state, the first latent intention, and the second latent intention. The first latent fused state may then be utilized as the first latent state z, and the first latent intention and the second latent intention may be utilized as the first intention w.

320 320 320 1,t 1,t 2,t 2 In some embodiments, the second agentB may decode the first latent state zand the first latent intention winto the first environment representation data and the first waypoint data. The second agentB may then combine the first environment representation data and the second sensory data to determine a second fused sensory data. The second agentB may then encode the second fused sensory data, the first waypoint data, and the second waypoint data to obtain a second latent fused state, the first latent intention, and the second latent intention. The second latent fused state may then be utilized as the second latent state z, and the first latent intention and the second latent intention may be utilized as the first intention w.

320 320 320 320 320 1,t 1,t 1,t 1,t 1,t 1,t 1,t 1,t 1,t 1,t 1,t φ 1,t 1,t 1,t 1,t+1 1,t+1 The first agentA may then proceed to determine a first reward rby applying an augmented model p on the first latent state zand the first hidden state data h, and determine a first continuation by applying the augmented model p on the first latent state zand the first hidden state data h. The first agentA may then determine a first value vby applying a value function to the first latent state z, the first hidden state data h, and the first reward rto determine a first action a. The first agentA may then determine a next hidden state hby applying a sequence model fto the first latent state z, the first hidden state data h, and the first action a. The first agentA may then determine a first dynamic predictor {circumflex over (z)}based on the next hidden state h. This process may continue iteratively for up to k time intervals (t+k), enabling the first agentA to predict future states, evaluate rewards, and refine actions over an extended horizon to ensure optimal decision-making and adaptability to dynamic environments.

320 320 320 320 320 1,t φ 2,t 2,t φ 2,t 2,t 2,t 2,t 2,t 2,t 2,t 2,t+1 φ 2,t 2,t 2,t 2,t+1 2,t+1 The second agentB may then proceed to determine a first reward rby applying the augmented model pon the second latent state zand the second hidden state data h, and determine a second continuation by applying the augmented model pon the second latent state zand the second hidden state data h. The second agentB may then determine a second value vby applying a value function to the second latent state z, the second hidden state data h, and the second reward rto determine a second action a. The second agentB may then determine a subsequent hidden state hby applying the sequence model fto the second latent state z, the second hidden state h, and the second action a. The second agentB may then determine a second dynamic predictor zbased on the subsequent hidden state h. This process may continue iteratively for up to k time intervals (t+k), enabling the second agentB to predict future states, evaluate rewards, and refine actions over an extended horizon to ensure optimal decision-making and adaptability to dynamic environments.

3 FIG.C 3 FIG.B 3 FIG.C 330 330 i,t+k i,t+k φ φ i,t+k i,t+k In, an algorithmic representationfor determining a dynamic predictor zand a hidden state hfor up to k time intervals is shown. The algorithmic representationmay utilize the sequence model f, the augmented model p, the encoder and decoder, and other models to determine the dynamic predictor zand the hidden state h. Each agent i inmay also utilize the algorithm ofto determine the dynamic predictor and the hidden state iteratively for up to k time intervals.

4 FIG. 402 404 406 408 410 408 410 408 410 402 presents a visual output (e.g., Bird's Eye View (BEV)) generated by the model. The visual output may comprise a vehicle, a line of sight, a predicted path, identified objects, and non-identified object. The identified objectsrepresent objects that were recognized based on the vehicle's own sensory data and processing. In contrast, the non-identified objectsrepresent objects that were detected or inferred with the assistance of latent state and latent intention data shared by other agents. It should be appreciated that when predicting the state a plurality of future points in time, the position of the tracked objects,may shift based on the respectively predicted trajectories relative to the planned trajectory of the vehicle.

i,t i,t−k i,t−k k=0 i,t i,t−k i,t−k k=0 i,t i,t i,t K-1 K-1 402 To adaptively determine when and what information to share, each agent may continuously monitors its prediction performance by evaluating prediction errors. For agent i at time t, this may involve comparing previously predicted latent states and intentions {circumflex over (X)}={{circumflex over (z)}, ŵ}against the actual observations X={z, w}from the last K time steps. The prediction error may be calculated as E=∥{circumflex over (X)}−X∥. If this error exceeds a predefined threshold c, the agent may increase its communication range and exchanges relevant latent information. To increase the communication range, the vehiclemay increase a transmit power of a transceiver used to broadcast a polling request for latent information, switch communication modes to a longer range communication protocol (e.g., switching from Wi-Fi to LoRaWan), query a wider range of vehicle associated with locations maintained in a vehicle database to identify additional vehicles from which to request latent information, etc. This adaptive process ensures that only necessary information is shared, significantly reducing communication overhead while maintaining prediction accuracy. As a result, reducing the prediction error may contribute to minimizing the sub-optimality gap in the value function, thereby improving the overall performance of the multi-agent system.

5 FIG. 5 FIG. 502 502 502 502 502 502 displays a comprehensive flow diagram of a multi-agent sharing information to determine generalizations including path planning, decisions, and predictions.includes three agents, an ego vehicleA, a first vehicleB, and a second vehicleC. The ego vehicleA may use information from the first vehicleB and the second vehicleC to determine generalization.

502 504 502 504 502 504 Each agent may have its own observation data (or sensory data) collected using one or more sensors. The ego vehicleA may use a RGB camera and a LIDARA to collect ego observation data. The first vehicleB may use a RGB cameraB to collect first observation data. The second vehicleC may use a RGB camera and a radarC to collect second observation data.

502 502 502 206 506 502 506 502 506 2 FIG.B The ego vehicleA, upon collecting the ego observation data, may use the ego observation data to derive with ego waypoint data. The ego vehicleA may use the internal path-planning algorithms (e.g., software architecture of) on the ego observation data to derive with sequence of target points that the vehicle plans on travelling. The ego vehicleA may then use a bird's eye view generator (e.g., BEV generator moduleD) on the ego observation data and the ego waypoint data to generate an ego local bird's eye view (BEV)A. Similarly, the first vehicleB may use the internal path-planning algorithms using the first observation data to determine first waypoint data, and use the bird's eye view generator on the first observation data and the first waypoint data to generate a first BEVB. The second vehicleC may use the internal path-planning algorithms using the second observation data to determine second waypoint data, and use the bird's eye view generator on the second observation data and the second waypoint data to generate a second BEVC.

502 508 206 1 506 502 502 502 502 502 1,t 1 1,t 1 1,t 1,t 1,t 1,t 1,t 1 The first vehicleB may then use a first encoderB (e.g., encoder moduleA) to encode the first BEVB to determine a first latent state zand a first latent intention w. In some embodiments, the first vehicleB may encode the first observation data to determine the first latent state zand encode the first waypoint data to determine the first latent intention w. In some other embodiments, the first vehicleB may encode a first hidden state hand the first observation data to determine the first latent state z. The first hidden state hmay indicate contextual information about past observations or actions of the first vehicleB. For example, the first hidden state hmay comprise recent positions and speeds of nearby vehicles, a sequence of traffic light changes, or pedestrian movements observed over the last few seconds. The first vehicleB may then proceed to transmit the first latent state zand the first latent intention wto the ego vehicleA.

502 508 506 502 502 502 502 502 2,t 2 2,t 2 2,t 2,t 2,t 2,t 2 The second vehicleC may then use a second encoderC to encode the second BEVC to determine a second latent state zand a second latent intention w. In some embodiments, the second vehicleC may encode the second observation data to determine the second latent state zand encode the second waypoint data to determine the second latent intention w. In some other embodiments, the second vehicleC may encode a second hidden state hand the second observation data to determine the second latent state z. The second hidden state hmay indicate contextual information about past observations or actions of the second vehicleC. The second vehicleC may then proceed to transmit the second latent state zand the second latent intention wto the ego vehicleA.

502 512 502 506 502 502 506 1,t 1 1,t 1 2,t 2 2,t 2 The ego vehicleA, upon receiving the first latent state zand the first latent intention w, may use a decoderto decode the first latent state zinto the first environment representation data and the first latent intention winto the first waypoint data. The ego vehicleA may then use the bird's eye view generator to reconstruct the first BEVB view using the first environment representation data and the first waypoint data. The ego vehicleA, upon receiving the second latent state zand the second latent intention w, may decode the second latent state zinto the second environment representation data, and the second latent intention winto the second waypoint data. The ego vehicleA may then use the bird's eye view generator to reconstruct the second BEVC using the second environment representation data and the second waypoint data.

502 506 506 506 514 502 502 514 514 514 502 502 502 514 502 The ego vehicleA may combine the ego BEVA, the first BEVB, and the second BEVC to determine a fused BEV. In some embodiments, the ego vehicleA, instead of generating the first BEV and the second BEV, may generate the fused BEV based on the ego observation data, the first observation data, the second observation data, the ego waypoint data, the first waypoint data, and the second waypoint data. For example, the ego vehicleA may generate fused observation data (e.g., fused state data), and use the fused observation data along with the ego waypoint data, the first waypoint data, and the second waypoint data to generate the fused BEV. The fused BEVmay provide a more comprehensive view of the environment than any component BEV that has been fused into the fused BEV. The fused BEVmay map the ego waypoint data, first waypoint data, and the second waypoint data indicating predicted path of the ego vehicleA, the first vehicleB, and the second vehicleC to the fused BEV. This combined perspective may allow the ego vehicleA to understand the relative positions, trajectories, and intentions of nearby agents, as well as to identify potential obstacles and hazards in a broader spatial context.

502 During this information sharing process, the ego vehicleA may evaluate its prediction errors by comparing the previously predicted latent states against actual observations. If the prediction error exceeds a threshold, the ego vehicle may adjust its communication strategy, potentially increasing its communication range to acquire more information from additional agents. This prediction-accuracy-driven approach may ensure that information sharing occurs only when necessary to maintain accurate predictions, thereby reducing overall communication overhead while preserving prediction performance. This adaptive mechanism may enhance both partial observability (by obtaining only necessary information from other agents) and non-stationarity (by continuously updating and enhancing predictions based on the latest shared information).

502 514 514 518 502 520 502 524 502 524 522 3 FIG.B The ego vehicleA, upon determining the fused BEV, may encode the fused BEVusing an encoderto determine a latent fused state, latent intentions (including an ego latent intention, the first latent intention, and the second latent intention). Based on the latent fused state and the latent intentions, the ego vehicleA may derive generalizations, predicting latent statesof the ego vehicleA and actionsat different time intervals K. The ego vehicleA may determine the actionsby applying a memory augmented neural networkto the latent fused state and latent fused waypoint. Details relating to deriving generalizations can be found in.

502 502 514 In some embodiments, the ego vehicleA, upon receiving the first latent state, the first latent intention, the second latent state, and the second latent intention, may fuse the ego observation data, the first latent state, and the second latent state to determine a latent fused state. The ego vehicleA may then proceed to convert the latent fused state into a fused BEV.

6 FIG. 5 FIG. 6 FIG. 602 602 602 602 602 602 602 602 602 602 illustrates a multi-agent system for determining generalizations, where each agent has a different encoding model. Because each agent has a different encoding model, the other vehicles (e.g., a first vehicleB and a second vehicleC) are unable to share latent states and latent intentions to an ego vehicleA as illustrated in. Said another way, the ego vehicleA would not be able to decode the latent states and latent intentions of the other vehicles to obtain the observation data and waypoint data due to the differences in encoding. Therefore, the multi-agent system inmay not enjoy the benefit of low communication load when communicating with each other. However, the multi-agent system may still utilize the world model to derive quick generalizations. The multi-agent system may comprise the ego vehicleA, the first vehicleB, and the second vehicleC. The ego vehicleA may receive information from the first vehicleB and the second vehicleC.

602 604 602 604 602 604 Each agent may have its own observation data collected using one or more sensors. The ego vehicleA may use a RGB camera and a LIDARA to collect ego observation data. The first vehicleB may use a RGB cameraB to collect first observation data. The second vehicleC may use a RGB camera and a radarC to collect second observation data.

602 602 602 606 602 606 602 606 2 FIG.B The ego vehicleA, upon collecting the ego observation data, may use the ego observation data to derive with ego waypoint data. The ego vehicleA may use the internal path-planning algorithms () using the ego observation data to derive a sequence of target points that the vehicle plans on travelling. The ego vehicleA may then use a bird's eye view generator on the ego observation data and the ego waypoint data to generate an ego local bird's eye view (BEV)A. Similarly, the first vehicleB may use the internal path-planning algorithms using the first observation data to determine first waypoint data, and use the bird's eye view generator on the first observation data and the first waypoint data to generate a first BEVB. The second vehicleC may use the internal path-planning algorithms using the second observation data to determine second waypoint data, and use the bird's eye view generator on the second observation data and the second waypoint data to generate a second BEVC.

602 606 602 602 602 602 606 602 606 602 602 602 602 606 The first vehicleB may then proceed to transmit the first BEVB to the ego vehicleA. In some embodiments, the first vehicleB may transmit the first observation data and the first waypoint data to the ego vehicleA, and the ego vehicleA may proceed to determine the first BEVB using the BEV generator. The second vehicleC may proceed to transmit the second BEVC to the ego vehicleA. In some embodiments, the second vehicleC may transmit the second observation data and the second waypoint data to the ego vehicleA, and the ego vehicleA may proceed to determining the second BEVC using the BEV generator.

602 606 606 606 608 602 608 502 608 602 The ego vehicleA may combine the ego BEVA, the first BEVB, and the second BEVC to determine a fused BEV. In some embodiments, the ego vehicleA may generate the fused BEVusing the ego observation data, the first observation data, the second observation data, the ego waypoint data, the first waypoint data, and the second waypoint data. For example, the ego vehicleA may determine a fused observation data (e.g., fused state data), and use the fused observation data along with the ego waypoint data, the first waypoint data, and the second waypoint data to determine the fused BEV. The fused BEVmay provide a comprehensive view of the environment. This combined perspective may allow the ego vehicleA to understand the relative positions, trajectories, and intentions of nearby agents, as well as to identify potential obstacles and hazards in a broader spatial context.

602 608 608 610 602 612 602 616 602 616 614 3 FIG.B The ego vehicleA, upon determining the fused BEV, may encode the fused BEVusing an encoderto determine a latent fused state and latent intentions (including an ego latent intention, a first latent intention, and a second latent intention). Based on the latent fused state and the latent intentions, the ego vehicleA may derive generalizations, predicting latent statesof the ego vehicleA and actionsat different time intervals K. The ego vehicleA may determine the actionsby applying a memory augmented neural networkto the latent fused state and latent fused waypoint. Details to deriving generalizations can be found in.

7 FIG. 2 FIG. 700 200 depicts a computer-implemented method generating a trajectory prediction for a vehicle. Although the methodis described below with regard to a vehicle computing systemand components thereof as illustrated in, it will be understood that other similarly suitable devices and/or components may be used instead and that hardware systems, components, or infrastructure may be implemented.

702 200 At block, the vehicle computing systemmay obtain first sensory data and first waypoint data associated with a first vehicle. The first sensory data may be data collected from one or more sensors, such as camera images, LiDAR scans, radar data, or GPS information, describing the first vehicle's environment and surroundings. The first waypoint data may be a sequence of planned target points or coordinates that represent the intended path or trajectory of the first vehicle, including planned movements such as lane changes, speed adjustments, or stops over a given time horizon. In some embodiments, the first sensory data may represent a bird's-eye-view of an environment proximate to the first vehicle.

704 200 At block, the vehicle computing systemmay receive a latent state and a latent intention from a second vehicle. The latent state may represent encoded second sensory data associated with the second vehicle. The latent intention may represent encoded second waypoint data associated with the second vehicle. In some embodiments, the second vehicle may be located within a predefined distance threshold from the first vehicle. The predefined distance threshold may be based on a broadcasting signal range of the first vehicle, or a global positioning system (GPS) distance with respect to the first vehicle. In some embodiments, the latent state may include third sensory data associated with a third vehicle and the latent intention may include third waypoint data associated with the third vehicle. In some other embodiments, the latent state may have a smaller feature space than a second sensory data, and the latent intention may have a smaller feature space than a second waypoint data.

706 At block, the vehicle computing system may fuse the first sensory data and the latent state into a latent fused state. The latent fused state may represent a more comprehensive and accurate representation of the vehicle's environment. In some embodiments, the vehicle computing system may decode the latent state into environment representation data, and the latent intention into a second waypoint data. The vehicle computing system may then fuse the first sensory data and the environment representation data into a fused environment representation. The vehicle computing system may encode the fused environment representation into the latent fused state. In some other embodiments, the vehicle computing system may obtain first hidden state data associated with the first vehicle. The vehicle computing system may then fuse the first sensory data, the latent state, and the first hidden state into the latent fused state.

708 710 At block, the vehicle computing system may apply the latent fused state, the latent intention, and the first waypoint data into a route planning module to generate a trajectory prediction for the first vehicle. In some embodiments, the vehicle computing system may encode the first waypoint data. The vehicle computing system may then apply the latent fused state, the latent intention, and the encoded first waypoint data into the motion planning module to generate the trajectory prediction. In some other embodiments, the vehicle computing system may generate the trajectory prediction for a plurality of future time intervals. At block, the vehicle computing system may control the first vehicle based on the trajectory prediction.

In some embodiments, the vehicle computing system may execute the motion planning module on historical waypoint data and historical sensory data to generate the first waypoint data. By leveraging historical data, the motion planning module may generate a sequence of targe points that account for recuring obstacles, patterns, and behaviors. By leveraging historical waypoint data, the motion planning module may help with anticipating different scenarios and situations.

In some embodiments, the vehicle computing system may encode the first sensory data and the first waypoint data. The vehicle computing system may then transmit the encoded first sensory data and the encoded first waypoint data to the second vehicle. The motion planning module of the second vehicle may be configured to analyze the encoded first sensory data and the encoded first waypoint data to generate a trajectory prediction for the second vehicle. In further embodiments, the vehicle computing system may encode the first sensory data by defining a subset of the first sensory data based on an absolute position of the first vehicle to the second vehicle. The vehicle computing system may then encode the subset of the first sensory data without encoding other portions of the first sensor data.

In some embodiments, the vehicle computing system may determine that a prediction error for a past trajectory prediction is above a predefined threshold. The vehicle computing system may then expand a communication range of the one or more transceivers of the first vehicle. The vehicle computing system may then receive one or more latent states and latent intentions from one or more vehicles in the communication range.

Aspect 1. A method for generating a trajectory prediction for a vehicle, the method comprising: obtaining, by one or more processors, first sensory data and first waypoint data associated with a first vehicle; receiving, by the one or more processors, a latent state and a latent intention from a second vehicle, wherein the latent state represents encoded second sensory data associated with the second vehicle, wherein the latent intention represents encoded second waypoint data associated with the second vehicle; fusing, by the one or more processors, the first sensory data and the latent state into a latent fused state; applying, by the one or more processors, the latent fused state, the latent intention, and the first waypoint data into a motion planning module to generate a trajectory prediction for the first vehicle; and applying, by the one or more processors, the trajectory prediction to an autonomous control system to generate control commands for controlling the first vehicle.

Aspect 2. The method of aspect 1, wherein fusing the first sensory data and the latent state into the latent fused state further comprises: decoding, by the one or more processors, the latent state into environment representation data, and the latent intention into a second waypoint data; fusing, by the one or more processors, the first sensory data and the environment representation data into a fused environment representation; and encoding, by the one or more processors, the fused environment representation into the latent fused state.

Aspect 3. The method of aspect 2, wherein applying the latent fused state, the latent intention, and the first waypoint data into the motion planning module to generate the trajectory prediction for the first vehicle further comprises: encoding, by the one or more processors, the first waypoint data; and applying, by the one or more processors, the latent fused state, the latent intention, and the encoded first waypoint data into the motion planning module to generate the trajectory prediction

Aspect 4. The method of any one of aspects 1 to 3, wherein the latent state has a smaller feature space than a second sensory data.

Aspect 5. The method of either one of aspects 1 to 4, wherein the latent intention has a smaller feature space than a second waypoint data.

Aspect 6. The method of any one of aspects 1 to 5, wherein the second vehicle is located within a predefined distance threshold from the first vehicle.

Aspect 7. The method of aspect 6, wherein the predefined distance threshold is based on a broadcasting signal range of the first vehicle.

Aspect 8. The method of either one of aspects 6 or 7, wherein the predefined distance threshold is based on a global positioning system (GPS) distance with respect to the first vehicle.

Aspect 9. The method of any one of aspects 1 to 8, wherein generating the trajectory prediction comprises: generating, by the one or more processors, the trajectory prediction for a plurality of future time intervals.

Aspect 10. The method of any one of aspects 1 to 9, further comprising: executing, by the one or more processors, the motion planning module on historical waypoint data and historical sensory data to generate the first waypoint data.

Aspect 11. The method of any one of aspects 1 to 10, further comprising: encoding, by the one or more processors, the first sensory data and the first waypoint data; and transmitting, by the one or more processors, the encoded first sensory data and the encoded first waypoint data to the second vehicle, wherein a motion planning module of the second vehicle is configured to analyze the encoded first sensory data and the encoded first waypoint data to generate a trajectory prediction for the second vehicle.

Aspect 12. The method of aspect 11, wherein encoding the first sensory data comprises: defining, by the one or more processors, a subset of the first sensory data based on an absolute position of the first vehicle to the second vehicle; and encoding, by the one or more processors, the subset of the first sensory data without encoding other portions of the first sensory data.

Aspect 13. The method of any one of aspect 11 or 12, wherein the first vehicle comprises one or more transceivers, further comprising: determining, by the one or more processors, that a prediction error for a past trajectory prediction is above a predefined threshold; expanding, by the one or more processors, a communication range of the one or more transceivers of the first vehicle; and receiving, by the one or more processors, one or more latent states and latent intentions from one or more vehicles in the expanded communication range.

Aspect 14. The method of any one of aspects 1 to 13, wherein the latent state includes third sensory data associated with a third vehicle and the latent intention includes third waypoint data associated with the third vehicle.

Aspect 15. The method of any one of aspects 1 to 14, wherein the first sensory data represents a bird's-eye-view of an environment proximate to the first vehicle.

Aspect 16. The method of any one of aspects 1 to 15, wherein fusing the first sensory data and the latent state into the latent fused state further comprises: obtaining, by the one or more processors, first hidden state data associated with the first vehicle; and fusing, by the one or more processors, the first sensory data, the latent state, and the first hidden state into the latent fused state.

Aspect 17. A computer system for generating a trajectory prediction for a vehicle, comprising: one or more processors, and a tangible, non-transitory memory coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the computer system to: obtain first sensory data and first waypoint data associated with a first vehicle; receive a latent state and a latent intention from a second vehicle, wherein the latent state represents encoded second sensory data associated with the second vehicle, wherein the latent intention represents encoded second waypoint data associated with the second vehicle; fuse the first sensory data and the latent state into a latent fused state; apply the latent fused state, the latent intention, and the first waypoint data into a motion planning module to generate a trajectory prediction for the first vehicle; and apply the trajectory prediction to an autonomous control system to generate control commands for controlling the first vehicle.

2 16 Aspect 18. The computer system of aspect 17, wherein instructions that, when executed by the one or more processors, cause the computer system to perform the method of any one of claimsto.

Aspect 19. A non-transitory computer-readable medium storing executable instructions for generating a trajectory prediction for a vehicle that, when executed by one or more processors, causes the one or more processors to: obtain first sensory data and first waypoint data associated with a first vehicle; receive second sensory data and second waypoint data from a second vehicle; fuse the first sensory data and the second sensory data into a fused sensory data; encode the fused sensory data into a latent fused state, the first waypoint data, and the second waypoint data; apply the latent fused state, the encoded first waypoint data, and the encoded second waypoint data into a motion planning module to generate a trajectory prediction for the first vehicle; and apply the trajectory prediction to an autonomous control system to generate control commands for controlling the first vehicle.

2 16 Aspect 20. The non-transitory computer-readable medium of aspect 19, wherein the instructions further cause the one or more processors to perform the method of any one of claimsto.

Although the text herein sets forth a detailed description of numerous different embodiments, it should be understood that the legal scope of the invention is defined by the words of the claims set forth at the end of this patent. The detailed description is to be construed as exemplary only and does not describe every possible embodiment, as describing every possible embodiment would be impractical, if not impossible. One could implement numerous alternate embodiments, using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims.

It should also be understood that, unless a term is expressly defined in this patent using the sentence “As used herein, the term ‘______’ is hereby defined to mean . . . ” or a similar sentence, there is no intent to limit the meaning of that term, either expressly or by implication, beyond its plain or ordinary meaning, and such term should not be interpreted to be limited in scope based upon any statement made in any section of this patent (other than the language of the claims). To the extent that any term recited in the claims at the end of this disclosure is referred to in this disclosure in a manner consistent with a single meaning, that is done for sake of clarity only so as to not confuse the reader, and it is not intended that such claim term be limited, by implication or otherwise, to that single meaning. Finally, unless a claim element is defined by reciting the word “means” and a function without the recital of any structure, it is not intended that the scope of any claim element be interpreted based upon the application of 35 U.S.C. § 112(f).

Throughout this specification, plural instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. Structures and functionality presented as separate components in example configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.

Additionally, certain embodiments are described herein as including logic or a number of routines, subroutines, applications, or instructions. These may constitute either software (code embodied on a non-transitory, tangible machine-readable medium) or hardware. In hardware, the routines, etc., are tangible units capable of performing certain operations and may be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client or server computer system) or one or more modules of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a module that operates to perform certain operations as described herein.

In various embodiments, a module may be implemented mechanically or electronically. Accordingly, the term “module” should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. Considering embodiments in which modules are temporarily configured (e.g., programmed), each of the modules need not be configured or instantiated at any one instance in time. For example, where the modules comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respective different modules at different times. Software may accordingly configure a processor, for example, to constitute a particular module at one instance of time and to constitute a different module at a different instance of time.

Modules can provide information to, and receive information from, other modules. Accordingly, the described modules may be regarded as being communicatively coupled. Where multiple of such modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) that connect the modules. In embodiments in which multiple modules are configured or instantiated at different times, communications between such modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple modules have access. For example, one module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further module may then, at a later time, access the memory device to retrieve and process the stored output. Modules may also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information).

The various operations of example methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. The modules referred to herein may, in some example embodiments, comprise processor-implemented modules. Moreover, the systems and methods described herein are directed to an improvement to computer functionality and improve the functioning of conventional computers.

Similarly, the methods or routines described herein may be at least partially processor-implemented. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processor or processors may be located in a single location (e.g., within a home environment, an office environment or as a server farm), while in other embodiments the processors may be distributed across a number of locations.

The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the one or more processors or processor-implemented modules may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the one or more processors or processor-implemented modules may be distributed across a number of geographic locations.

Unless specifically stated otherwise, discussions herein using words such as “processing,” “computing,” “calculating,” “determining,” “presenting,” “displaying,” or the like may refer to actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information. Some embodiments may be described using the expression “coupled” and “connected” along with their derivatives. For example, some embodiments may be described using the term “coupled” to indicate that two or more elements are in direct physical or electrical contact. The term “coupled,” however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other. The embodiments are not limited in this context.

As used herein any reference to “one embodiment” or “an embodiment” means that a particular element, feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment. In addition, use of the “a” or “an” are employed to describe elements and components of the embodiments herein. This is done merely for convenience and to give a general sense of the description. This description, and the claims that follow, should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise.

As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

This detailed description is to be construed as exemplary only and does not describe every possible embodiment, as describing every possible embodiment would be impractical, if not impossible. One could implement numerous alternate embodiments, using either current technology or technology developed after the filing date of this application. Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for systems and methods for autonomous vehicle services and operations through the disclosed principles herein. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.

The particular features, structures, or characteristics of any specific embodiment may be combined in any suitable manner and in any suitable combination with one or more other embodiments, including the use of selected features without corresponding use of other features. In addition, many modifications may be made to adapt a particular application, situation or material to the essential scope and spirit of the present invention. It is to be understood that other variations and modifications of the embodiments of the present invention described and illustrated herein are possible in light of the teachings herein and are to be considered part of the spirit and scope of the present invention.

While the preferred embodiments of the invention have been described, it should be understood that the invention is not so limited and modifications may be made without departing from the invention. The scope of the invention is defined by the appended claims, and all devices that come within the meaning of the claims, either literally or by equivalence, are intended to be embraced therein. It is therefore intended that the foregoing detailed description be regarded as illustrative rather than limiting, and that it be understood that it is the following claims, including all equivalents, that are intended to define the spirit and scope of this invention.