Method and System for Real-Time Three-Dimensional Scene Graph Creation

The information provided in this section is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

The present disclosure relates generally to real-time three-dimensional scene graph creation. In the realm of autonomous vehicle technology, scene graphs are pivotal for enabling vehicles to perceive and interact with their surroundings. Currently, scene graphs are generated using a combination of sensor data from cameras, light detection and ranging (LiDAR), radar, and ultrasonic sensors. These sensors provide a representation of the environment of the vehicle, which is then processed to identify objects, their positions, and their movements. This information is used by the vehicle's autonomous systems to make decisions about navigation and maneuvering. However, the existing methods primarily focus on two-dimensional data.

Despite advancements in sensor technology, current scene graph construction methods lack the integration of three-dimensional information predictions. Three-dimensional predictions of scene graphs may enable the vehicle to plan more precise trajectories, especially in complex driving scenarios. For instance, three-dimensional scene graphs are essential for capturing the spatial relationships and depth information necessary for accurate vehicle trajectory and maneuver planning. With this crucial data, autonomous vehicles may truly understand a scene to navigate safely and efficiently, particularly in environments with varying elevations, obstacles, and dynamic elements.

One aspect of the disclosure provides a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations that include receiving, as input to a transformer model, sensor data corresponding to a two-dimensional (2D) image of a current scene of a vehicle, and encoding, using a birds-eye view encoder, the sensor data to generate a corresponding sequence of feature embeddings. Here, the sequence of feature embeddings corresponds to a three-dimensional (3D) representation of the current scene of the vehicle. The operations also include decoding the sequence of feature embeddings using two or more feature-specific decoders executing in parallel, and processing, using a prediction network, the decoded sequence of feature embeddings to convert the decoded sequence feature of embeddings into semantic features. The operations further include processing, using a topology network, the semantic features and the decoded sequence of feature embeddings to generate an adjacency matrix representing a 3D view of the current scene of the vehicle.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the two or more feature-specific decoders executing in parallel each include a plurality of transformer layers. In these implementations, each transformer layer may include a cross-attention head. Additionally or alternatively, decoding the sequence of feature embeddings using the two or more feature-specific decoders executing in parallel may include executing cross-attention of the sequence of feature embeddings between corresponding transformer layers of the two or more feature-specific decoders.

In some examples, the sensor data includes a set of image frames. In these examples, the operations may further include, for each image frame of the set of image frames, extracting feature embeddings of the current scene. Here, encoding, using the birds-eye view encoder, the sensor data to generate the corresponding sequence of feature embeddings may include projecting the sensor data into the corresponding sequence of feature embeddings. In some implementations, the prediction network includes a multilayer perceptron network. In some examples, the 2D image of the current scene of the vehicle includes at least two elements. In these examples, the adjacency matrix may predict a strength of the relationship between the at least two elements.

Another aspect of the disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed by the data processing hardware cause the data processing hardware to perform operations that include receiving, as input to a transformer model, sensor data corresponding to a two-dimensional (2D) image of a current scene of a vehicle, and encoding, using a birds-eye view encoder, the sensor data to generate a corresponding sequence of feature embeddings. Here, the sequence of feature embeddings corresponds to a three-dimensional (3D) representation of the current scene of the vehicle. The operations also include decoding the sequence of feature embeddings using two or more feature-specific decoders executing in parallel, and processing, using a prediction network, the decoded sequence of feature embeddings to convert the decoded sequence feature of embeddings into semantic features. The operations further include processing, using a topology network, the semantic features and the decoded sequence of feature embeddings to generate an adjacency matrix representing a 3D view of the current scene of the vehicle.

This aspect may include one or more of the following optional features. In some implementations, the two or more feature-specific decoders executing in parallel each include a plurality of transformer layers. In these implementations, each transformer layer may include a cross-attention head. Additionally or alternatively, decoding the sequence of feature embeddings using the two or more feature-specific decoders executing in parallel may include executing cross-attention of the sequence of feature embeddings between corresponding transformer layers of the two or more feature-specific decoders.

In some examples, the sensor data includes a set of image frames. In these examples, the operations may further include, for each image frame of the set of image frames, extracting feature embeddings of the current scene. Here, encoding, using the birds-eye view encoder, the sensor data to generate the corresponding sequence of feature embeddings may include projecting the sensor data into the corresponding sequence of feature embeddings. In some examples, the 2D image of the current scene of the vehicle includes at least two elements. In these examples, the adjacency matrix may predict a strength of the relationship between the at least two elements.

Another aspect of the disclosure provides a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations that include receiving, as input to a transformer model, sensor data corresponding to a two-dimensional (2D) image of a current scene of a vehicle, the 2D image including at least two elements. The operations also include encoding, using a birds-eye view encoder, the sensor data to generate a corresponding sequence of feature embeddings. Here, the sequence of feature embeddings corresponds to a three-dimensional (3D) representation of the current scene of the vehicle. The operations further include decoding the sequence of feature embeddings using two or more feature-specific decoders executing in parallel, and processing, using a topology network, the decoded sequence of feature embeddings to generate an adjacency matrix representing a 3D view of the current scene of the vehicle. Here, the adjacency matrix predicts a strength of the relationship between the at least two elements.

The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

Corresponding reference numerals indicate corresponding parts throughout the drawings.

Example configurations will now be described more fully with reference to the accompanying drawings. Example configurations are provided so that this disclosure will be thorough, and will fully convey the scope of the disclosure to those of ordinary skill in the art. Specific details are set forth such as examples of specific components, devices, and methods, to provide a thorough understanding of configurations of the present disclosure. It will be apparent to those of ordinary skill in the art that specific details need not be employed, that example configurations may be embodied in many different forms, and that the specific details and the example configurations should not be construed to limit the scope of the disclosure.

The terminology used herein is for the purpose of describing particular exemplary configurations only and is not intended to be limiting. As used herein, the singular articles “a,” “an,” and “the” may be intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms “comprises,” “comprising,” “including,” and “having,” are inclusive and therefore specify the presence of features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order discussed or illustrated, unless specifically identified as an order of performance. Additional or alternative steps may be employed.

When an element or layer is referred to as being “on,” “engaged to,” “connected to,” “attached to,” or “coupled to” another element or layer, it may be directly on, engaged, connected, attached, or coupled to the other element or layer, or intervening elements or layers may be present. In contrast, when an element is referred to as being “directly on,” “directly engaged to,” “directly connected to,” “directly attached to,” or “directly coupled to” another element or layer, there may be no intervening elements or layers present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., “between” versus “directly between,” “adjacent” versus “directly adjacent,” etc.). As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.

The terms “first,” “second,” “third,” etc. may be used herein to describe various elements, components, regions, layers and/or sections. These elements, components, regions, layers and/or sections should not be limited by these terms. These terms may be only used to distinguish one element, component, region, layer or section from another region, layer or section. Terms such as “first,” “second,” and other numerical terms do not imply a sequence or order unless clearly indicated by the context. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the example configurations.

In this application, including the definitions below, the term “module” may be replaced with the term “circuit.” The term “module” may refer to, be part of, or include an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog/digital discrete circuit; a digital, analog, or mixed analog/digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; memory (shared, dedicated, or group) that stores code executed by a processor; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.

The term “code,” as used above, may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, and/or objects. The term “shared processor” encompasses a single processor that executes some or all code from multiple modules. The term “group processor” encompasses a processor that, in combination with additional processors, executes some or all code from one or more modules. The term “shared memory” encompasses a single memory that stores some or all code from multiple modules. The term “group memory” encompasses a memory that, in combination with additional memories, stores some or all code from one or more modules. The term “memory” may be a subset of the term “computer-readable medium.” The term “computer-readable medium” does not encompass transitory electrical and electromagnetic signals propagating through a medium, and may therefore be considered tangible and non-transitory memory. Non-limiting examples of a non-transitory memory include a tangible computer readable medium including a nonvolatile memory, magnetic storage, and optical storage.

The apparatuses and methods described in this application may be partially or fully implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions that are stored on at least one non-transitory tangible computer readable medium. The computer programs may also include and/or rely on stored data.

A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an “application,” an “app,” or a “program.” Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

The non-transitory memory may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. The non-transitory memory may be volatile and/or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.

These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer readable medium, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

Various implementations of the systems and techniques described herein can be realized in digital electronic and/or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

The processes and logic flows described in this specification can be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user and optionally a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

1 FIG. 100 10 60 40 40 10 60 10 40 100 10 60 Referring to, in some implementations, a systemincludes a vehiclein communication with a remote systemvia a network. The networkmay include a wireless local area network (WLAN) that facilitates communication and interoperability between the vehicleand the remote systemwithin an environment of the vehicle. Thus, the networkcan include Wireless Fidelity (WiFi®) (e.g., IEEE 802.11), Low-Rate Wireless Personal Area Networks (e.g., IEEE 802.15.4), worldwide interoperability for microwave access (WiMAX), 3G, 4G, Long Term Evolution (LTE), 5G, digital subscriber line (DSL), Bluetooth, Near Field Communication (NFC), or any other wireless standards, or Ethernet (e.g., IEEE 802.3). The systemmay additionally include one or more access points (AP) (not shown) configured to facilitate wireless communication between the vehicleand the remote system.

10 60 200 18 102 10 302 102 102 10 10 230 200 212 212 230 200 262 102 10 262 102 2 FIG. 3 FIG.A As shown, the vehicleand/or the remote systemexecute a three-dimensional (3D) graph system() configured to receive sensor dataof a sceneof the vehicle, infer one or more elements() in the sceneand their relationships simultaneously, and generate/predict a fully 3D scene graph of the scenesurrounding the vehiclefor downstream applications of the vehicle. As described in further detail below, two or more decodersof the 3D graph systemthat execute in parallel may pull/share feature embeddingsvia a cross-attention mechanism. By executing tasks in parallel, this sharing of feature embeddingsprovides synergy between the two or more decodersthat not only improves performance of the 3D graph systemover existing perception systems but allows for the real-time generation of an adjacency matrixrepresenting the 3D scene graph of the scenesurrounding the vehicle. Advantageously, the construction of the adjacency matrixallows downstream applications to more clearly understand the scenefor safer and more accurate maneuvers, particularly in autonomous driving modes.

200 10 200 200 10 10 10 12 14 12 12 In the example shown, the 3D graph systemis implemented within the vehicle. However, the 3D graph systemcan be implemented in any other propulsion system, such as, without limitation, motorcycles, trucks, off-road vehicles, farm equipment, trains, aircraft, and the like. Additionally, while the 3D graph systemis shown implemented within a vehicle, it can be implemented on other computing devices (e.g., computing devices in communication with the vehicle), such as, without limitation, a smart phone, tablet, smart display, desktop/laptop, smart watch, smart appliance, or smart glasses/headset. The vehicleincludes data processing hardwareand memory hardwarestoring instructions that when executed on the data processing hardwarecause the data processing hardwareto perform operations.

1 2 FIGS.and 1 FIG. 10 18 16 16 10 16 16 10 16 10 As shown in, the vehicleis configured to receive sensor datadetected/captured by a sensor system. The sensor systemmay include one or more of cameras, a forward collision mitigation system, radio detection and ranging (RADAR), light detection and ranging (LIDAR) capable of capturing image data, and other external sensors of the vehicle. While the sensor systemshown inis disposed on a front side of the vehicle, it should be appreciated that the sensor systemmay include sensors located throughout the vehicle. For example, the sensor systemmay provide 360-degree surround sensing of an environment of the vehicle.

18 104 102 10 104 102 104 302 104 302 104 10 102 302 302 The sensor datamay include one or more image framesof the scenelocated outside of the vehicle. Notably, the one or more image framesof the sceneare two-dimensional (2D). These image framesmay capture one or more elementswithin the image frame. As used herein, elementsmay generally refer to dynamic elements such as pedestrians and cars, as well as static elements such as traffic lights, road signs, road markings, etc. While the image framesare 2D, the downstream applications of the vehiclebenefit from 3D representations of the scene, such as the position and orientation of each element, as well as the relationship between elementsin order to fully plan for vehicle maneuvers.

60 62 64 62 62 200 10 60 3 200 10 60 202 18 104 302 262 302 102 10 1 4 FIGS.- The remote system(e.g., server, cloud computing environment) also includes data processing hardwareand memory hardwarestoring instructions that when executed on the data processing hardwarecause the data processing hardwareto perform operations. In some examples, execution of the 3D graph systemis shared across the vehicleand the remote system. As described in greater detail with respect to, theD graph systemexecuting on the vehicleand/or the remote systemexecutes a transformer modelthat is configured to receive sensor dataincluding the image framescapturing the one or more elementsrepresented in 2D and generate an adjacency matrixrepresenting a 3D view of the elementsin the current sceneof the vehicle.

2 FIG. 1 FIG. 3 200 202 202 210 230 230 240 240 250 260 202 200 220 212 1 102 10 14 64 a e y Referring to, theD scene graph systemexecuting the transformer modelis shown. The transformer modelis a deep neural network (DNN) that includes a birds-eye view (BEV) encoder, a plurality of feature specific decoders,-configured to execute/process in parallel, a prediction head(also referred to as a prediction network), a feature transformation module, and a topology network. Additionally, the transformer modelof the 3D scene graph systemhas access to a memory buffer. The memory buffer may include previously generated feature embeddings-corresponding to previous scenesencountered by the vehicle, and may be stored in the memory hardware,of.

200 18 302 16 302 18 210 18 18 212 302 18 18 104 104 18 210 102 10 210 18 212 18 212 210 212 212 y−1 As shown, the 3D scene graph systemcontinuously receives/processes the sensor dataincluding the one or more elementsdetected by the sensor systemto identify one or more elementsin the sensor data. The BEV encoderreceives, as input, the 2D sensor dataand encodes the sensor datato generate, as output, a sequence of feature embeddingsrepresenting the elementsin the sensor data. As described above, the sensor datamay include a set of image frames. Here, for each image frameof the sensor data, the BEV encodermay extract feature embeddings for the current sceneof the vehicle. In these instances, the BEV encodermay encode the sensor datato generate the corresponding sequence of feature embeddingsby projecting the sensor datainto the corresponding sequence of feature embeddings. In some instances, the BEV encodergenerates the sequence of feature embeddingsbased on the previous time-step of the sequence of feature embeddings.

200 212 220 230 230 212 232 230 230 230 230 230 302 230 230 230 18 302 230 230 230 200 a e a e a b c d e Thereafter, the 3D graph systemmay store the sequence of feature embeddingsin the memory buffer. Additionally, each of the feature-specific decoders-receives, as input, the sequence of feature embeddings, and generates, as output a decoded sequence of feature embeddings. As shown, the feature-specific decoders-include a 3D sign decoder, a 3D vectorized map decoder, a 3D actor decoder(i.e., dynamic elements), a 3D traffic light decoder, and a 3D road marking decoder. Each feature-specific decodermay be specifically trained to extract specific features from sensor datathat correlate to a specific category such as, without limitation, signs, maps (e.g., topography of a road), dynamic elements, traffic lights, road markings, etc. It should be understood that, although five (5) feature-specific decodersare shown, the disclosure contemplates that more decoders, or fewer decodersmay also be used to implement the 3D graph system.

4 FIG. 4 FIG. 230 230 230 230 230 230 230 230 212 212 230 230 230 230 230 212 212 230 212 230 230 230 212 212 230 230 230 230 212 212 212 212 230 230 a e a e a e a e a e a a b b a b a b a b a b a b a b a e Referring briefly to. Each of the feature-specific decoders-is shown, where each feature-specific decoderis communicatively coupled with each of the other feature-specific decoders. One or more of the feature-specific decoders-may include a multilayer perceptron (MLP) network. In some instances, each feature-specific decodermay include a plurality of transformer layers. Here, each transformer layer of each respective feature-specific decodermay include at least one cross-attention head configured to perform inter-feature information passing of the respective feature embeddings-between the other feature-specific decoders-. For example, at each time-step of processing by each of the feature-specific decoders-, each feature-specific decodermay determine the positional estimates of the sequence of feature embeddings, and when the sequence of feature embeddingsof a first feature specific decoderare within a threshold distance of the sequence of feature embeddingsof a second-feature specific decoder, the feature-specific decoders,may execute cross-attention of the sequence of feature embeddings,between the transformer layers of the first feature-specific decoderand the second feature-specific decoder. Thereafter, the feature-specific decoders,may perform channel-wise concatenation to concatenate the sequence of feature embeddings,and pass the concatenated sequence of feature embeddings,to a feed-forward network for the next time-step of processing. As should be apparent from, at each time-step, the feature-specific decoders-may execute this cross-attention mechanism between one another in parallel.

2 FIG. 240 232 232 232 232 232 232 242 242 302 102 240 240 242 302 302 102 250 232 232 232 232 232 232 252 240 250 a e a e a e a e Referring again to, the prediction headreceives, as input, the decoded sequence of feature embeddings-output from the feature-specific decoders-and processes the decoded sequence of feature embeddingsto convert the decoded sequence of feature embeddingsinto semantic features. For instance, the semantic featuresmay include one or more of the category, width, height, position (e.g., xyz coordinates), and orientation of each of the one or more elementsin the scene. In some cases, the prediction headincludes an MLP network. For example, both a regressor and classifier of the prediction headmay be each implemented as an MLP network, where the regressor MLP network is configured to generate a regression prediction indicating the semantic featuresof the position and orientation of each element, while the classifier MLP network is configured to generate a classification prediction of the type (e.g., dynamic elements such as pedestrians and cars, as well as static elements such as traffic lights, road signs, road markings) of elementis present in the scene. Similarly, the feature transformation modulereceives, as input, the decoded sequence of feature embeddings-output from the feature-specific decoders-and processes the decoded sequence of feature embeddingsto convert the decoded sequence of feature embeddingsinto transformed features. Like the prediction head, the feature transformation modulemay be implemented as an MLP network.

260 232 242 252 262 102 250 232 242 252 302 102 260 302 302 102 302 262 102 10 262 302 102 10 260 302 102 Thereafter, the topology networkmay receive, as input, one or more of the decoded sequence of feature embeddings, the semantic features, and the transformed features, and generate, as output, the adjacency matrixrepresenting a 3D view of the current scene. Here, the topology networkis trained to process the input feature embeddings, the semantic features, and the transformed featuresto determine which elementspresent in the sceneare strongly connected semantically. For instance, the topology networkmay predict a confidence for the relationship between each elementand the other elementsin the sceneand, based on the predicted confidences of each pair of elements, generate the adjacency matrixrepresenting the 3D view of the current sceneof the vehicle. In other words, the adjacency matrixpredicts the strength of the relationship between at least two elementsin the sceneof the vehicle. Here, the topology networkmay infer topological information about the 3D view of the current scene based on the predicted confidences of the associations between each pair of elementsin the scene.

3 3 FIGS.A andB 3 FIG.A 3 FIG.B 102 262 1012 104 102 10 104 302 302 200 102 302 302 302 302 302 302 302 302 3021 302 302 302 302 302 302 200 104 102 262 262 302 302 302 302 10 302 262 302 302 10 302 302 262 10 10 a o a g h i j b f c d e o k m n a o a o a o a o Referring to, example views of a 2D sceneand its corresponding adjacency matrixof the 3D view of the scene, are shown respectively. With particular reference to, an image frameof the current sceneof the vehicleis shown in 2D. In the image frame, a plurality of elements-are shown. In particular, the 3D graph systemmay identify that the sceneincludes the dynamic elements,,,, andcorresponding to cars, static elements,corresponding to traffic lights, static elements,corresponding to road signs, static elements,,corresponding to lanes, and static elements,,corresponding to road markings. After the 3D graph systemprocesses the image frameof the current scene, it generates, as output, the adjacency matrixshown in. Here, the adjacency matrixincludes each of the same elements-, but represents each of the elements-as a birds-eye-view 3D perspective around the vehicle. In addition to the strength of the relationship between pairs of elements, the adjacency matrixmay also include the relative size, location, and orientation of each of the elements-with respect to the vehicleand to the other elements-. Notably, the adjacency matrixmay transmitted to downstream applications of the vehicle(e.g., steering control, brake control, etc.) for planning maneuvers of the vehicleas it drives along the road.

5 FIG. 1 4 FIGS.- 1 FIG. 1 FIG. 500 500 12 62 14 64 500 includes a flowchart of an example arrangement of operations for a methodfor real-time three-dimensional (3D) scene graph creation. The methodmay be described with reference to. Data processing hardware (e.g., data processing hardware,of) may execute instructions stored on memory hardware (e.g., memory hardware,of) to perform the example arrangement of operations for the method.

502 500 202 18 102 10 504 500 210 18 212 212 102 10 At operation, the methodincludes receiving, as input to a transformer model, sensor datacorresponding to a two-dimensional (2D) image of a current sceneof a vehicle. At operation, the methodincludes encoding, using a birds-eye view encoder, the sensor datato generate a corresponding sequence of feature embeddings. Here, the corresponding sequence of feature embeddingscorrespond to a three-dimensional (3D) representation of the current sceneof the vehicle.

506 500 212 230 230 500 508 240 232 232 242 510 500 260 242 232 262 102 10 a e At operation, the methodalso includes decoding the sequence of feature embeddingsusing two or more feature specific decoders,-executing in parallel. The methodalso includes, at operation, processing, using a prediction network, the decoded sequence of feature embeddingsto convert the decoded sequence of feature embeddingsinto semantic features. At operation, the methodfurther includes processing, using a topology network, the semantic featuresand the decoded sequence of feature embeddingsto generate an adjacency matrixrepresenting a 3D view of the current sceneof the vehicle.

6 FIG. 1 4 FIGS.- 1 FIG. 1 FIG. 600 600 12 62 14 64 600 includes a flowchart of an example arrangement of operations for a methodfor real-time three-dimensional (3D) scene graph creation. The methodmay be described with reference to. Data processing hardware (e.g., data processing hardware,of) may execute instructions stored on memory hardware (e.g., memory hardware,of) to perform the example arrangement of operations for the method.

600 602 202 18 102 10 302 604 600 210 18 212 212 102 10 The methodincludes, at operation, receiving, as input to a transformer model, sensor datacorresponding to a two-dimensional (2D) image of a current sceneof a vehicle. Here, the 2D image includes at least two elements. At operation, the methodincludes encoding, using a birds-eye view encoder, the sensor datato generate a corresponding sequence of feature embeddings. Here, the corresponding sequence of feature embeddingscorrespond to a three-dimensional (3D) representation of the current sceneof the vehicle.

606 600 212 230 230 608 600 260 232 262 102 10 262 302 a e At operation, the methodfurther includes decoding the sequence of feature embeddingsusing two or more feature specific decoders,-executing in parallel. At operation, the methodincludes processing, using a topology network, the decoded sequence of feature embeddingsto generate an adjacency matrixrepresenting a 3D view of the current sceneof the vehicle. Here, the adjacency matrixpredicts a strength of the relationship between the at least two elements.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.

The foregoing description has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular configuration are generally not limited to that particular configuration, but, where applicable, are interchangeable and can be used in a selected configuration, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.