Supplementing Top-Down Predictions with Image Features

2. The system of claim 1, wherein the image data comprises one or more of a position or orientation of an image sensor relative to the autonomous vehicle used to capture an image included in the image data.

3. The system of claim 1, wherein the concatenated representation comprises a first feature included in the image feature representation in a first channel and a second feature from the top-down representation in a second channel of a multi-channel image.

4. The system of claim 1, wherein the additional sensor data comprises one or more of radar data, lidar data, time of flight data, or map data.

6. The method of claim 5, wherein receiving the predicted behavior comprises receiving a prediction probability associated with the predicted behavior.

7. The method of claim 5, wherein the first portion of the machine-learned model comprises a neural network encoder and a neural network decoder.

9. The method of claim 8, wherein the first image is captured at a first time and the second image is captured at a second time after the first time.

10. The method of claim 9, wherein the first camera and the second camera are a same camera.

11. The method of claim 5, wherein the first data comprises a first multi-channel representation and the second data comprises a second multi-channel representation.

12. The method of claim 5, wherein the object is a vehicle, and the predicted behavior of the vehicle comprises one or more of a stay-in-lane action, a turn action, or a lane change action.

14. The one or more computer-readable media of claim 13, wherein receiving the predicted behavior comprises receiving a prediction probability associated with the predicted behavior.

15. The one or more computer-readable media of claim 13, wherein the first portion of the machine-learned model comprises a neural network encoder and a neural network decoder.

17. The one or more computer-readable media of claim 16, wherein the first image is captured at a first time and the second image is captured at a second time after the first time.

18. The one or more computer-readable media of claim 17, wherein the first camera and the second camera are a same camera.

19. The one or more computer-readable media of claim 13, wherein the first data comprises a first multi-channel representation and the second data comprises a second multi-channel representation.

20. The one or more computer-readable media of claim 13, wherein the predicted behavior comprises an image representation of behavior, a pixel of the image representation of behavior associated with a probability the object will occupy a corresponding position in the environment.