Systems, methods and apparatuses of lidar sensors of autonomous vehicles. A lidar sensor can include: a memory configured to store a lidar image and an Artificial Neural Network (ANN); an inference engine configured to use the (ANN) to analyze the lidar image and generate inference results; and a communication interface coupled to a computer system of a vehicle to implement an advanced driver assistance system to operate the controls according to the inference results and a sensor data stream generated by sensors configured on the vehicle.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
2. The lidar sensor of claim 1, wherein the beam steerer is a solid-state device having no movable mechanical components.
A lidar sensor system includes a beam steerer that directs laser beams without mechanical movement. The beam steerer is a solid-state device, eliminating the need for moving parts, which reduces wear and improves reliability. The system emits laser pulses toward a target, receives reflected signals, and processes the data to determine distance and other properties of the target. The solid-state beam steerer uses optical elements such as microelectromechanical systems (MEMS), liquid crystal arrays, or other non-mechanical methods to steer the laser beam. This design enhances durability, reduces maintenance, and improves performance in harsh environments. The lidar sensor may be used in autonomous vehicles, robotics, industrial automation, or other applications requiring precise distance measurements without mechanical complexity. The solid-state approach also allows for faster beam steering and higher resolution compared to traditional mechanical systems. The system may include additional components such as a laser source, detector, and signal processing unit to analyze the reflected signals and generate distance data. The absence of moving parts in the beam steerer ensures long-term stability and accuracy in various operating conditions.
3. The lidar sensor of claim 2, wherein the artificial neural network includes a spiking neural network, or a deep neural network, or any combination thereof.
This invention relates to lidar sensor systems incorporating advanced neural network architectures for improved signal processing and object detection. The lidar sensor is designed to address challenges in accurately interpreting complex environmental data, particularly in autonomous vehicle applications where real-time, high-resolution 3D mapping is critical. The sensor utilizes an artificial neural network to process raw lidar point cloud data, enhancing detection accuracy and reducing computational overhead. The neural network may be configured as a spiking neural network, which mimics biological neural processes for energy-efficient computation, or a deep neural network, which leverages multiple layers to extract hierarchical features from the data. Alternatively, the system may combine both architectures to optimize performance. The neural network processes the lidar data to identify and classify objects, distinguish between static and dynamic elements, and generate precise spatial maps. This approach improves the sensor's ability to handle noisy or occluded data, ensuring reliable operation in diverse environments. The integration of these neural network types allows the lidar system to adapt to varying conditions, such as weather interference or sensor degradation, while maintaining high accuracy and low latency. The overall system enhances autonomous navigation by providing more robust and interpretable 3D environmental data.
4. The lidar sensor of claim 2, wherein the lidar sensor is configured to generate lidar images at a rate and resolution that exceeds a communication bandwidth between the host interface and the computer system of the vehicle.
A lidar sensor system for autonomous vehicles addresses the challenge of processing high-resolution lidar data in real-time. The system includes a lidar sensor that captures detailed 3D images of the vehicle's surroundings at a high frame rate. The sensor is designed to generate lidar images with a resolution and update rate that surpasses the communication bandwidth available between the sensor's host interface and the vehicle's onboard computer system. This ensures that the sensor can provide high-fidelity environmental data without being limited by data transmission constraints. The system may also include a data processing unit that pre-processes the lidar data to reduce the volume of information sent to the vehicle's central computer, optimizing bandwidth usage while maintaining critical data integrity. The lidar sensor's high-resolution output enables precise object detection, mapping, and navigation, which are essential for autonomous driving. The system ensures that the vehicle's computer receives timely and accurate lidar data, even when the communication link has limited capacity. This approach enhances the reliability and performance of autonomous vehicle systems by mitigating data bottlenecks.
5. The lidar sensor of claim 2, wherein the attributes include reflection intensity of the points, range to the points, and direction to the points.
A lidar sensor system captures and processes point cloud data from a surrounding environment. The system includes a lidar sensor configured to emit light pulses and detect reflections to determine attributes of points in the environment. These attributes include the reflection intensity of the points, the range (distance) to the points, and the direction (angular position) to the points. The sensor generates a point cloud dataset comprising these attributes for each detected point. The system may further process this data to filter, classify, or analyze the points for applications such as obstacle detection, mapping, or navigation. The reflection intensity provides information about surface properties, while range and direction enable spatial localization. The sensor may be integrated into autonomous vehicles, robotics, or other systems requiring environmental perception. The system may also include additional components, such as a processor, to further analyze the point cloud data for specific use cases. The attributes are used to distinguish between different types of surfaces, objects, or environmental features, improving the accuracy and reliability of the lidar-based perception system.
7. The vehicle of claim 6, wherein the inference results identify an object that reflects lidar signals in the lidar image.
A system for object detection in autonomous vehicles uses lidar sensors to capture lidar images of the vehicle's surroundings. The system processes these images to identify objects that reflect lidar signals, distinguishing them from other environmental features. The lidar sensor emits light pulses and measures their reflections to generate a 3D point cloud, which is then analyzed to detect objects of interest. The system includes a neural network trained to classify these reflections, distinguishing between relevant objects (e.g., vehicles, pedestrians) and irrelevant reflections (e.g., road markings, vegetation). The neural network outputs inference results that highlight objects with significant lidar signal reflections, improving detection accuracy in varying environmental conditions. This approach enhances the vehicle's ability to navigate safely by accurately identifying obstacles and potential hazards in real-time. The system may also integrate additional sensors, such as cameras or radar, to supplement lidar data and refine object detection. The neural network is optimized for low-latency processing, ensuring timely decision-making for autonomous driving functions. The overall system improves situational awareness and reduces false positives in object detection, supporting safer and more reliable autonomous vehicle operation.
8. The vehicle of claim 7, wherein the inference results include a portion extracted from the lidar image using the artificial neural network and a control signal for operating or driving the vehicle.
This invention relates to autonomous vehicle systems that use lidar imaging and artificial neural networks for perception and control. The system processes lidar data to generate inference results, which include both extracted features from the lidar image and control signals for vehicle operation. The artificial neural network analyzes the lidar data to identify relevant portions of the image, such as obstacles, road boundaries, or other objects, and outputs these extracted features. Additionally, the system generates control signals based on the neural network's analysis to autonomously operate or drive the vehicle, such as adjusting steering, acceleration, or braking. The integration of lidar-based perception with direct control signal generation enables real-time decision-making for autonomous navigation. The system may also include other components, such as sensors for additional data collection, processing units for executing the neural network, and actuators for implementing the control signals. This approach improves the vehicle's ability to interpret its environment and respond dynamically to ensure safe and efficient autonomous operation.
9. The vehicle of claim 6, wherein a portion of the sensor data stream is provided to the lidar sensor to implement sensor fusion based on the lidar image and the portion of the sensor data stream.
This invention relates to vehicle sensor systems, specifically improving perception and decision-making by integrating data from multiple sensors, including lidar and other sources. The problem addressed is the need for accurate and reliable environmental perception in autonomous or semi-autonomous vehicles, where sensor fusion enhances detection and tracking of objects. The vehicle includes a lidar sensor that generates a lidar image of the environment. Additionally, the vehicle has one or more other sensors, such as cameras or radar, that produce a sensor data stream. A portion of this sensor data stream is provided to the lidar sensor to enable sensor fusion. The lidar sensor processes the lidar image in combination with the portion of the sensor data stream, allowing for improved object detection, classification, and tracking. This fusion enhances the accuracy and robustness of the vehicle's perception system by leveraging complementary data from different sensors. The system may also include a processor that further analyzes the fused data to support navigation, obstacle avoidance, and other autonomous driving functions. The invention ensures that the vehicle can operate safely and efficiently in dynamic environments by integrating multiple sensor inputs into a cohesive perception framework.
10. The vehicle of claim 6, wherein the inference engine includes a memristor crossbar array configured to store parameters of the artificial neural network and to perform matrix multiplication and accumulation operations in the artificial neural network.
This invention relates to a vehicle equipped with an artificial neural network (ANN) inference engine that leverages a memristor crossbar array for efficient computation. The system addresses the challenge of performing real-time, energy-efficient neural network operations in automotive applications, where low latency and high computational efficiency are critical. The vehicle includes an inference engine designed to execute artificial neural networks, with the engine incorporating a memristor crossbar array. This array is used to store the parameters (weights and biases) of the neural network and to perform matrix multiplication and accumulation operations, which are fundamental to neural network computations. Memristor crossbar arrays enable in-memory computing, reducing data movement between memory and processing units, thereby improving energy efficiency and speed. The memristor crossbar array is structured to allow parallel processing of multiple data points simultaneously, accelerating the inference process. This architecture is particularly beneficial for tasks such as object detection, path planning, and sensor fusion in autonomous or semi-autonomous vehicles, where rapid decision-making is essential. The use of memristors also reduces the physical footprint and power consumption compared to traditional digital processors, making it suitable for embedded automotive systems. The system may integrate with other vehicle components, such as cameras, LiDAR, or radar sensors, to process sensor data in real time. The memristor-based inference engine enhances the vehicle's ability to perform complex computations efficiently, supporting advanced driver-assistance systems (ADAS) and autonomous driving functionalities.
12. The method of claim 11, wherein the analyzing includes determining an identification or a classification of an object captured in the lidar image.
This invention relates to lidar-based object detection and classification in autonomous systems. The technology addresses the challenge of accurately identifying and categorizing objects in lidar-generated point cloud data, which is critical for applications like autonomous vehicles, robotics, and environmental monitoring. The method involves processing lidar images to extract features and analyze them to determine the identity or classification of objects within the scene. This includes distinguishing between different types of objects, such as vehicles, pedestrians, or static obstacles, based on their spatial and reflective characteristics. The analysis may involve comparing extracted features against a database of known object profiles or using machine learning models trained to recognize specific object classes. The system may also incorporate temporal data, such as object movement patterns, to improve classification accuracy. By accurately identifying and classifying objects in real-time, the method enhances the reliability of autonomous navigation and decision-making systems. The invention is particularly useful in dynamic environments where rapid and precise object recognition is essential for safety and efficiency.
13. The method of claim 12, wherein the analyzing includes extracting, from the lidar image, a portion showing the object.
This invention relates to object detection and analysis using lidar (light detection and ranging) imaging systems. The technology addresses the challenge of accurately identifying and extracting objects from lidar-generated point clouds or images, which is critical for applications such as autonomous vehicles, robotics, and environmental monitoring. Traditional methods often struggle with noise, occlusion, or complex backgrounds, leading to inaccurate object detection. The method involves analyzing lidar data to detect and isolate objects within the scanned environment. A lidar system captures a three-dimensional point cloud or image representing the surrounding scene. The analysis step includes extracting a specific portion of the lidar image that corresponds to a detected object, effectively isolating the object from the background or other elements in the scene. This extraction process may involve segmentation techniques, clustering algorithms, or machine learning models trained to recognize object boundaries within the lidar data. By focusing on the extracted portion, subsequent processing steps, such as classification or tracking, can be performed with higher accuracy and efficiency. The method ensures that only relevant data associated with the object is processed, reducing computational overhead and improving detection performance. This approach is particularly useful in dynamic environments where real-time object recognition is required, such as in autonomous driving systems or industrial automation. The extracted object data can then be used for further analysis, such as determining the object's position, velocity, or type, enabling more informed decision-making in automated systems.
15. The method of claim 11, wherein the analyzing includes performing matrix multiplication and accumulation operations using a memristor crossbar array configured to store parameters of the artificial neural network.
This invention relates to artificial neural network (ANN) processing using memristor crossbar arrays. The technology addresses the computational inefficiency and energy consumption of traditional digital hardware in performing matrix multiplication and accumulation operations, which are fundamental to ANN computations. Memristor crossbar arrays offer a more efficient alternative by leveraging their analog, non-volatile memory properties to store and process ANN parameters in a single step, reducing latency and power consumption. The method involves analyzing input data by performing matrix multiplication and accumulation operations directly within a memristor crossbar array. The crossbar array is configured to store the parameters of the artificial neural network, such as weights and biases, in its memristive devices. During operation, input signals are applied to the array, and the memristor devices perform parallel, analog computations to produce the desired output. This approach eliminates the need for separate memory and processing units, enabling in-memory computing and improving overall system efficiency. The memristor crossbar array is designed to handle the high-dimensional matrix operations required for neural network inference or training. The analog nature of memristors allows for continuous weight updates, which is particularly useful for adaptive learning tasks. The method may also include techniques to mitigate issues like device variability and noise, ensuring reliable performance. This innovation is applicable in edge computing, AI accelerators, and other domains where low-power, high-speed neural network processing is critical.
16. The vehicle of claim 6, wherein the memory and the inference engine are configured on at least one first integrated circuit die; and the lidar sensor includes a light sensor configured on a second integrated circuit die.
This invention relates to an integrated vehicle system combining lidar sensing and processing capabilities on separate but interconnected integrated circuit (IC) dies. The system addresses the challenge of efficiently processing lidar data by co-locating the processing components with the sensing hardware while maintaining modularity. The vehicle includes a lidar sensor with a light sensor configured on a second IC die, enabling high-resolution optical detection. The memory and inference engine, responsible for storing and analyzing lidar data, are configured on at least one first IC die. This separation allows for optimized performance, where the sensing and processing components can be independently scaled or upgraded. The system may also include additional components such as a processor, communication interfaces, and power management circuits, all integrated into the vehicle's architecture. The design improves data processing efficiency by reducing latency between sensing and computation, while the modular approach enhances flexibility in system design and manufacturing. The invention is particularly useful in autonomous vehicles or advanced driver-assistance systems (ADAS) requiring real-time environmental perception.
17. The vehicle of claim 16, wherein the at least one first integrated circuit die and the second integrated circuit die are enclosed within a same integrated circuit package.
18. The vehicle of claim 17, wherein the memory and the inference engine in the at least one first integrated circuit die and the light sensor in the second integrated circuit die are connected via at least through-silicon vias.
This invention relates to an integrated circuit system for vehicle applications, particularly for processing sensor data with low latency and high efficiency. The system addresses the challenge of integrating multiple components, such as light sensors and processing units, into a compact and high-performance architecture. The invention includes at least two integrated circuit dies: a first die containing a memory and an inference engine for real-time data processing, and a second die containing a light sensor for capturing environmental or vehicle-related data. The memory stores data and instructions, while the inference engine performs computations, such as machine learning-based analysis, on the sensor data. The light sensor detects light intensity, color, or other optical properties relevant to vehicle operations, such as autonomous driving or driver assistance. The first and second dies are interconnected using through-silicon vias (TSVs), which enable high-speed, low-latency communication between the components. This integration reduces the need for external wiring, improves signal integrity, and enhances overall system performance. The system may be used in applications requiring rapid sensor data processing, such as obstacle detection, lane tracking, or adaptive lighting control in vehicles. The use of TSVs ensures efficient data transfer between the sensor and processing units, minimizing delays and improving real-time decision-making capabilities.
19. The vehicle of claim 18, wherein the memory and the inference engine are configured on at least one first integrated circuit die; and the light sensor is configured on a second integrated circuit die.
This invention relates to a vehicle equipped with an advanced driver assistance system (ADAS) that integrates a light sensor and a processing system for real-time environmental analysis. The system addresses the challenge of accurately detecting and interpreting light conditions to enhance vehicle safety and autonomy. The vehicle includes a light sensor that captures environmental light data, such as ambient brightness or specific light sources, and a processing system that analyzes this data to assist in navigation, obstacle detection, or other ADAS functions. The processing system comprises a memory for storing data and an inference engine for executing machine learning models or algorithms to interpret the sensor inputs. To optimize performance and reduce latency, the memory and inference engine are implemented on a first integrated circuit (IC) die, while the light sensor is fabricated on a separate second IC die. This modular design allows for specialized optimization of each component, improving efficiency and scalability. The system may further include additional sensors or processing units to enhance functionality, such as combining light data with other sensor inputs for comprehensive environmental awareness. The invention aims to provide a robust, high-performance solution for vehicles requiring precise light-based decision-making.
20. The method of claim 11, wherein the memory and the inference engine are configured on at least one first integrated circuit die; the lidar sensor includes a light sensor configured on a second integrated circuit die; the at least one first integrated circuit die and the second integrated circuit die are enclosed within a same integrated circuit package; and wherein the memory and the inference engine in the at least one first integrated circuit die and the light sensor in the second integrated circuit die are connected via at least through-silicon vias.
This invention relates to an integrated circuit system for lidar (light detection and ranging) applications, addressing the challenge of combining lidar sensing with processing capabilities in a compact, high-performance package. The system integrates a lidar sensor, a memory, and an inference engine within a single integrated circuit package. The lidar sensor includes a light sensor configured on a second integrated circuit die, while the memory and inference engine are configured on at least one first integrated circuit die. Both dies are enclosed within the same package, enabling efficient data processing and inference directly at the sensor level. The components are interconnected via through-silicon vias (TSVs), which facilitate high-speed, low-latency communication between the light sensor and the processing elements. This integration reduces the need for external connections, improves energy efficiency, and enhances system reliability by minimizing signal degradation. The system is particularly useful in applications requiring real-time lidar data processing, such as autonomous vehicles, robotics, and industrial automation, where compactness and performance are critical. The use of TSVs ensures robust electrical connections while maintaining a small form factor, making the system suitable for deployment in space-constrained environments.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
February 14, 2020
April 2, 2024
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.