A sensor unit includes a sensor interface, host interface, and pre-processing hardware. The sensor interface is coupled to a plurality of cameras configured to capture images around an autonomous driving vehicle (ADV). The host interface is coupled to a perception and planning system. The pre-processing hardware is coupled to the sensor interface to receive images from the plurality of cameras and to perform one or more pre-processing functions on the images and to transmit pre-processed images to the perception and planning system via the host interface. The perception and planning system is configured to perceive a driving environment surrounding the ADV based on the pre-processed images and to plan a path to control the ADV to navigate through the driving environment. The pre-processing functions can adjust for different calibrations and formats across the plurality of cameras.
Legal claims defining the scope of protection, as filed with the USPTO.
1. A sensor unit, comprising: a sensor interface to be coupled to a plurality of cameras configured to capture images around an autonomous driving vehicle (ADV); a host interface to be coupled to a perception and planning system; and pre-processing hardware including a field programmable gate array (FPGA), programmable logic array (PLA), application specific integrated circuit (ASIC) or hard-wired circuitry coupled to the sensor interface to receive images from the plurality of cameras and to perform one or more pre-processing functions on the images and to transmit pre-processed images not being processed by a central processing unit (CPU) or graphics processing unit (GPU) to the perception and planning system via the host interface, wherein the perception and planning system is configured to perceive a driving environment surrounding the ADV based on the pre-processed images and to plan a path to control the ADV to navigate through the driving environment.
2. The sensor unit of claim 1 , wherein the pre-processing functions adjusts for different calibrations and formats across the plurality of cameras.
3. The sensor unit of claim 1 , wherein the pre-processing hardware performs the one or more pre-processing functions not being performed by the plurality of cameras.
4. The sensor unit of claim 3 , wherein the one or more pre-processing functions includes region of interest (ROI) image processing, image format conversion, image correction, or image high dynamic range (HDR) processing.
5. The sensor unit of claim 4 , wherein the HDR pre-processing function is not performed by the plurality of cameras.
6. The sensor unit of claim 1 , wherein the pre-processing hardware includes hardware for a plurality of channels to perform one or more pre-processing functions on captured images for each channel.
7. The sensor unit of claim 6 , wherein the pre-processing hardware includes a scheduler to schedule performing pre-processing functions across the plurality of channels.
8. A computer-implemented method for pre-processing images for autonomous driving, the method comprising: receiving, at a sensor unit via a sensor interface, a plurality of images from a plurality of cameras around an autonomous driving vehicle (ADV); performing one or more pre-processing functions on the images by pre-processing hardware including a field programmable gate array (FPGA), programmable logic array (PLA), application specific integrated circuit (ASIC) or hard-wired circuitry; and transmitting pre-processed images not being processed by a central processing unit (CPU) or graphics processing unit (GPU) to a perception and planning system via a host interface, wherein the perception and planning system is configured to perceive a driving environment surrounding the ADV based on at least the pre-processed images and to plan a path to control the ADV to navigate through the driving environment.
9. The method of claim 8 , wherein the pre-processing functions adjusts for different calibrations and formats across the plurality of cameras.
10. The method of claim 8 , wherein the pre-processing hardware performs the one or more pre-processing functions not being performed by the plurality of cameras.
11. The method of claim 10 , wherein the one or more pre-processing functions includes region of interest (ROI) image processing, image format conversion, image correction, or image high dynamic range (HDR) processing.
12. The method of claim 11 , wherein the HDR pre-processing function is not performed by the plurality of cameras.
13. The method of claim 8 , wherein the pre-processing hardware includes hardware for a plurality of channels to perform one or more pre-processing functions on captured images for each channel.
14. The method of claim 13 , wherein the pre-processing hardware includes a scheduler to schedule performing pre-processing functions across the plurality of channels.
15. An autonomous driving system, comprising: a plurality of sensors, the sensors including one or more cameras to capture images around an autonomous driving vehicle (ADV); a perception and planning system; and a sensor unit, wherein the sensor unit includes a sensor interface coupled to the sensors, a host interface coupled to the perception and planning system, and pre-processing hardware including a field programmable gate array (FPGA), programmable logic array (PLA), application specific integrated circuit (ASIC) or hard-wired circuitry coupled to the sensor interface to receive images from the cameras, to perform one or more pre-processing functions on the images, and to transmit pre-processed images not being processed by a central processing unit (CPU) or graphics processing unit (GPU) to the perception and planning system via the host interface, wherein the perception and planning system is configured to perceive a driving environment surrounding the ADV based on at least the pre-processed images and to plan a path to control the ADV to navigate through the driving environment.
16. The system of claim 15 , wherein the pre-processing functions adjusts for different calibrations and formats across the plurality of cameras.
17. The system of claim 15 , wherein the pre-processing hardware performs the one or more pre-processing functions not being performed by the plurality of cameras.
18. The system of claim 17 , wherein the one or more pre-processing functions includes region of interest (ROI) image processing, image format conversion, image correction, or image high dynamic range (HDR) processing.
19. The system of claim 18 , wherein the HDR pre-processing function is not performed by the plurality of cameras.
20. The system of claim 15 , wherein the pre-processing hardware includes hardware for a plurality of channels to perform one or more pre-processing functions on captured images for each channel.
21. The system of claim 20 , wherein the pre-processing hardware includes a scheduler to schedule performing pre-processing functions across the plurality of channels.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
April 30, 2019
November 23, 2021
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.