A method of exception handing for an autonomous vehicle (AV) includes identifying an exception situation; identifying relevant sensors for the exception situation; identifying relevant tools to the exception situation, the relevant tools usable by a tele-operator to resolve the exception situation; and presenting, on a display of the tele-operator, data from the relevant sensors and the relevant tools.
Legal claims defining the scope of protection. Each claim is shown in both the original legal language and a plain English translation.
1. A method of exception handing for an autonomous vehicle (AV), comprising: identifying an exception situation; identifying relevant sensors for the exception situation; identifying relevant tools to the exception situation, the relevant tools usable by a tele-operator to resolve the exception situation, wherein in a case that the exception situation is identified as a passenger requesting assistance, a first relevant tool is usable by the tele-operator to activate at least one of a speaker or microphone of the AV, wherein in a case that the exception situation is identified as an obstruction situation, a second relevant tool is a path-drawing tool, and wherein in a case that a proposed trajectory is received from the AV, a third relevant tool comprises an action usable by the tele-operator to accept the proposed trajectory; and presenting, on a display of the tele-operator, data from the relevant sensors and the relevant tools.
Autonomous vehicles (AVs) require robust exception handling to manage unexpected situations, such as passenger requests, obstructions, or trajectory planning. This invention addresses the need for efficient remote assistance by a tele-operator to resolve such exceptions. The method involves identifying an exception situation, such as a passenger needing help, an obstruction blocking the vehicle, or a proposed trajectory requiring review. Once the exception is detected, the system identifies the relevant sensors (e.g., cameras, LiDAR) to gather necessary data. It also determines the appropriate tools for the tele-operator to use. For passenger assistance, the system provides tools to activate the AV’s speaker or microphone, allowing two-way communication. If an obstruction is detected, a path-drawing tool enables the tele-operator to suggest alternative routes. When the AV proposes a trajectory, the tele-operator can accept or modify it using provided tools. The system then displays the sensor data and relevant tools on the tele-operator’s interface, enabling real-time intervention. This approach enhances AV safety and operational efficiency by leveraging remote human oversight for complex or ambiguous scenarios.
2. The method of claim 1 , wherein the exception situation is identified as an inside-vehicle exception situation, and wherein the relevant sensors comprise an inside-facing camera of the AV.
Autonomous vehicles (AVs) rely on sensor data to navigate and make decisions, but detecting and responding to exceptions inside the vehicle—such as passenger behavior or internal system malfunctions—remains challenging. This invention addresses this by using an inside-facing camera to identify and classify exception situations within the vehicle. The system processes data from the camera to detect anomalies, such as unusual passenger movements, objects obstructing the driver's view, or system failures like malfunctioning displays. Once an exception is identified, the system triggers appropriate responses, such as alerting the passenger, adjusting vehicle settings, or transmitting data to a remote monitoring system. The inside-facing camera provides real-time visual input, enabling the AV to assess the situation and take corrective action. This improves safety and reliability by ensuring internal conditions do not compromise the vehicle's operation or passenger well-being. The method integrates seamlessly with the AV's existing sensor network, enhancing situational awareness without requiring additional hardware. By focusing on inside-vehicle exceptions, the invention complements external sensor-based systems, providing a comprehensive approach to autonomous vehicle safety.
3. The method of claim 1 , wherein the exception situation is identified as an outside-vehicle exception situation, and wherein the relevant sensors comprise an outside-facing camera of the AV.
Autonomous vehicles (AVs) rely on sensor data to navigate and make decisions, but external factors like weather, obstructions, or sensor malfunctions can create exceptions that disrupt operation. This invention addresses the challenge of detecting and responding to such outside-vehicle exception situations by leveraging an outside-facing camera to identify and analyze potential issues. The method involves using the AV's outside-facing camera to monitor the environment for conditions that could impair sensor functionality or vehicle operation. For example, the camera may detect heavy rain, fog, or debris obstructing the view of other sensors like LiDAR or radar. Once an exception situation is identified, the system can trigger corrective actions, such as adjusting sensor parameters, rerouting the vehicle, or alerting the operator. The camera's visual data is processed to determine the severity and type of exception, ensuring the AV responds appropriately to maintain safety and functionality. By integrating the outside-facing camera as a primary tool for detecting external exceptions, the invention enhances the AV's ability to operate reliably in diverse and unpredictable environments. This approach reduces reliance on individual sensors and improves overall system robustness.
4. The method of claim 1 , further comprising: receiving an assistance request to resolve the exception situation from the AV.
Autonomous vehicles (AVs) often encounter exception situations where their onboard systems cannot determine a safe path forward, requiring human intervention. This invention addresses the need for efficient and reliable remote assistance to resolve such situations. The method involves detecting an exception situation in an AV, where the AV's autonomous driving system cannot proceed safely without external input. The AV then transmits data about the exception situation, including sensor data, vehicle state, and environmental conditions, to a remote assistance system. The remote assistance system processes this data to identify potential solutions and provides guidance to the AV, such as navigation instructions or control adjustments, to resolve the exception. The AV then executes the received guidance to proceed safely. The invention further includes receiving an assistance request from the AV to resolve the exception, ensuring that the remote system is actively engaged in providing timely support. This method improves AV safety and reliability by leveraging human expertise when automated systems are insufficient.
5. The method of claim 1 , wherein the relevant sensors comprise an off-board sensor.
A system and method for monitoring and analyzing vehicle performance using a combination of on-board and off-board sensors. The invention addresses the challenge of obtaining comprehensive and accurate vehicle data by integrating multiple sensor sources to improve diagnostics, maintenance, and operational efficiency. The system includes a vehicle equipped with on-board sensors that collect real-time data such as speed, engine performance, and environmental conditions. Additionally, off-board sensors, such as external cameras, radar, or other remote monitoring devices, provide supplementary data to enhance situational awareness and accuracy. The collected data is processed and analyzed to detect anomalies, predict maintenance needs, and optimize vehicle performance. The off-board sensors may be mounted on infrastructure, other vehicles, or stationary monitoring stations, ensuring broader coverage and more reliable data collection. This hybrid sensor approach improves diagnostic accuracy, reduces downtime, and enhances safety by providing a more complete picture of vehicle conditions and environmental factors. The system is particularly useful for fleet management, autonomous vehicles, and industrial applications where precise monitoring is critical.
6. The method of claim 5 , wherein the off-board sensor is an infrastructure sensor.
This invention relates to a method for using off-board sensors, specifically infrastructure sensors, to enhance the operation of a vehicle or system. The method involves collecting data from an infrastructure sensor, which is a sensor located outside the vehicle or system being monitored. This sensor is part of the surrounding environment or infrastructure, such as traffic monitoring systems, roadside sensors, or other fixed installations. The collected data is then processed to provide information that improves the performance, safety, or efficiency of the vehicle or system. The infrastructure sensor may detect environmental conditions, traffic patterns, or other relevant parameters that influence the vehicle's operation. By integrating this external sensor data, the method enables real-time adjustments or optimizations, such as route planning, collision avoidance, or energy management. The use of infrastructure sensors allows for broader and more accurate data collection compared to on-board sensors alone, leading to better decision-making and system responsiveness. This approach is particularly useful in autonomous vehicles, smart transportation networks, or industrial systems where external environmental factors play a critical role in performance. The method ensures that the vehicle or system adapts dynamically to changing conditions, improving overall functionality and reliability.
7. The method of claim 5 , wherein the off-board sensor is an agent sensor.
The invention relates to a system for monitoring and managing agents, such as chemical or biological agents, using off-board sensors. The system addresses the challenge of accurately detecting and tracking agents in environments where traditional on-board sensors may be limited in range, accuracy, or accessibility. The off-board sensor, which is an agent sensor, is positioned remotely from the primary monitoring device to provide enhanced detection capabilities. This sensor is designed to capture data related to the presence, concentration, or behavior of the agent in the surrounding environment. The system processes this data to generate insights, such as agent distribution, movement patterns, or potential risks, which can be used for safety, security, or operational decision-making. The off-board sensor may be deployed in fixed or mobile configurations, depending on the application, and can communicate wirelessly with the central monitoring system. This approach improves the reliability and coverage of agent detection, enabling more effective monitoring and control in various industrial, environmental, or security contexts.
8. The method of claim 1 , wherein identifying the relevant sensors comprising: identifying an infrastructure sensor that is within a geofence of the AV.
Autonomous vehicles (AVs) rely on sensor data to navigate and make decisions, but processing data from all available sensors can be computationally expensive and inefficient. This invention addresses the problem by selectively identifying and utilizing only the most relevant sensors for a given task, improving efficiency and performance. The method involves determining which sensors are relevant to the AV's current operations. This includes identifying infrastructure sensors, such as traffic lights, roadside units, or other fixed sensors, that are located within a predefined geofence around the AV. The geofence defines a spatial boundary to filter sensors based on proximity, ensuring only nearby infrastructure sensors are considered. By focusing on sensors within this boundary, the system reduces unnecessary data processing and enhances real-time decision-making. The method may also involve analyzing sensor data to assess relevance, such as determining whether a sensor provides critical information for navigation, obstacle detection, or traffic management. This selective approach optimizes computational resources and improves the AV's responsiveness to dynamic environments. The invention ensures that only the most pertinent sensor data is used, enhancing efficiency without compromising safety or functionality.
9. The method of claim 1 , wherein identifying the relevant sensors comprising: analyzing a first quality of a first image from an on-board sensor; analyzing a second quality of a second image from an off-board sensor; and in response to comparing the first quality to the second quality, identifying the off-board sensor to the tele-operator.
This invention relates to sensor selection for tele-operation systems, addressing the challenge of determining the most reliable sensor data source for remote operators. The method involves evaluating image quality from multiple sensors, including both on-board (mounted on the tele-operated device) and off-board (external) sensors, to dynamically select the best sensor for the tele-operator. The process includes analyzing the quality of images from an on-board sensor and an off-board sensor, then comparing these qualities. Based on this comparison, the system identifies and presents the off-board sensor to the tele-operator if its image quality is superior. This ensures the tele-operator receives the highest-quality visual data for effective remote control. The method may also involve similar evaluations for other sensors, ensuring continuous optimization of sensor selection based on real-time quality assessments. The invention improves tele-operation accuracy and reliability by dynamically adapting to changing environmental conditions and sensor performance.
10. A system for exception handing for an autonomous vehicle (AV), comprising: a memory; and a processor, the processor is configured to execute instructions stored in the memory to: identify an exception situation; identify a relevant sensor for the exception situation by steps including: responsive to first images from an a first sensor being more saturated with light than second images from a second sensor, identify the second sensor as the relevant sensor; identify relevant tools to the exception situation, the relevant tools usable by a tele-operator to resolve the exception situation; and present, on a display of the tele-operator, data from the relevant sensor and the relevant tools.
The system addresses exception handling in autonomous vehicles (AVs) by dynamically selecting the most suitable sensor data and tools for tele-operators to resolve unexpected situations. In AVs, sensors may capture images with varying light saturation levels, which can affect their reliability. The system identifies an exception situation and determines the most relevant sensor by comparing image saturation levels from multiple sensors. If images from a first sensor are more saturated than those from a second sensor, the second sensor is deemed more reliable and its data is prioritized. The system then identifies tools specific to the exception situation, which tele-operators can use to intervene and resolve the issue. The relevant sensor data and tools are displayed to the tele-operator, enabling efficient decision-making. This approach ensures that tele-operators receive the most accurate and useful information during critical situations, improving the AV's ability to handle exceptions effectively. The system enhances safety and operational reliability by leveraging adaptive sensor selection and tailored tele-operation tools.
11. The system of claim 10 , wherein the exception situation is identified as an inside-vehicle exception situation, and wherein the relevant sensor is an inside-facing camera of the AV.
The system relates to autonomous vehicles (AVs) and addresses the challenge of detecting and responding to exception situations inside the vehicle. The invention involves a sensor-based monitoring system that identifies abnormal conditions within the AV cabin, such as passenger misbehavior, medical emergencies, or other irregularities. The system uses an inside-facing camera to capture real-time visual data of the vehicle interior. Advanced image processing and machine learning algorithms analyze the camera feed to detect anomalies, such as unusual movements, distress signals, or unauthorized actions. Upon detecting an exception, the system triggers appropriate responses, which may include alerting the passenger, contacting emergency services, or adjusting the vehicle's operation to ensure safety. The system integrates with the AV's broader sensor network and decision-making framework to provide a comprehensive safety solution. By continuously monitoring the interior environment, the invention enhances passenger safety and vehicle security, addressing gaps in traditional AV monitoring systems that primarily focus on external conditions. The technology is particularly valuable for improving the reliability and trustworthiness of autonomous transportation.
12. The system of claim 10 , wherein the exception situation is identified as an outside-vehicle exception situation, and wherein the relevant sensor is an outside-facing camera of the AV.
Autonomous vehicles (AVs) rely on sensor data to navigate and make decisions, but external factors like weather, obstructions, or sensor malfunctions can create exceptions that disrupt operation. This invention addresses the challenge of detecting and responding to such exceptions by using an outside-facing camera to identify external conditions that may impair the AV's ability to operate safely. The system monitors the camera feed for anomalies, such as poor visibility, debris, or sensor failures, and classifies these as "outside-vehicle exception situations." Once detected, the system can trigger corrective actions, such as adjusting navigation, alerting the vehicle's control system, or requesting human intervention. The invention ensures that the AV can recognize and mitigate risks posed by external environmental factors, improving safety and reliability. The outside-facing camera serves as a critical sensor for identifying these exceptions, allowing the AV to adapt its behavior in real time. This approach enhances the vehicle's ability to handle unpredictable conditions, reducing the likelihood of accidents or operational failures.
13. The system of claim 10 , wherein the instructions further comprise instructions to: receive an assistance request to resolve the exception situation from the AV.
The system is designed for autonomous vehicle (AV) operations, specifically addressing the challenge of handling exception situations where the AV encounters conditions beyond its autonomous capabilities. The system includes a remote assistance platform that enables human operators to provide real-time support to the AV when it encounters such exceptions. The AV is equipped with sensors and communication modules to detect and report these situations, which may include complex driving scenarios, unexpected obstacles, or sensor failures. The remote assistance platform receives these reports and connects the AV with a human operator who can assess the situation and provide guidance. The operator may intervene by taking partial or full control of the AV, adjusting navigation parameters, or offering instructions to resolve the issue. The system ensures seamless communication between the AV and the operator, allowing for timely and effective resolution of exceptions. This approach enhances the safety and reliability of autonomous vehicle operations by leveraging human expertise when automated systems are insufficient. The system also includes features for logging and analyzing exception situations to improve future autonomous decision-making.
14. The system of claim 10 , wherein the second sensor is an off-board sensor.
A system for monitoring and managing vehicle operations includes a first sensor mounted on the vehicle to detect a physical parameter, such as tire pressure or temperature, and a second sensor that is located off-board the vehicle. The off-board sensor is positioned externally to the vehicle and communicates with the on-board sensor to provide additional data for analysis. The system processes the combined data from both sensors to generate insights or alerts related to vehicle performance, safety, or maintenance needs. The off-board sensor may be part of a roadside monitoring station, a mobile device, or another external system, enabling real-time or periodic data collection. This setup enhances accuracy and reliability by cross-referencing on-board and off-board measurements, reducing false readings and improving decision-making for vehicle operators or fleet managers. The system may also include a communication module to transmit data to a central server or user interface for further analysis and reporting. The off-board sensor's external placement allows for broader environmental monitoring, such as detecting road conditions or external hazards that could impact vehicle operations. This dual-sensor approach ensures comprehensive monitoring and proactive maintenance, improving overall vehicle safety and efficiency.
15. The system of claim 14 , wherein the off-board sensor is an infrastructure sensor.
The system involves a vehicle monitoring and control system that uses off-board sensors to detect and respond to environmental conditions. The system is designed to improve vehicle safety and efficiency by integrating data from external sensors that are not physically attached to the vehicle. These sensors provide real-time information about the vehicle's surroundings, such as road conditions, weather, or traffic patterns, which the vehicle's onboard systems use to adjust operations. The system processes the sensor data to generate control signals that modify vehicle behavior, such as adjusting speed, braking, or steering, to avoid hazards or optimize performance. The system also includes a communication module that transmits the sensor data and control signals between the vehicle and the off-board sensors, ensuring seamless data exchange. In this specific embodiment, the off-board sensor is an infrastructure sensor, meaning it is part of the roadway or surrounding infrastructure, such as traffic lights, roadside cameras, or embedded road sensors, providing fixed-point data to enhance situational awareness. The system dynamically adapts to the environment by continuously analyzing the infrastructure sensor data and updating vehicle controls accordingly. This approach reduces reliance on onboard sensors alone, improving accuracy and reliability in monitoring and responding to external conditions.
16. The system of claim 14 , wherein the off-board sensor is an agent sensor.
The system relates to a vehicle monitoring and control system that utilizes off-board sensors to enhance situational awareness and decision-making. The system addresses the challenge of limited on-board sensor capabilities by integrating external sensors to provide additional data for vehicle operations. The off-board sensor, which is an agent sensor, is deployed outside the vehicle to collect environmental or operational data that supplements the vehicle's on-board sensors. This agent sensor may include devices such as cameras, radar, lidar, or other detection systems that monitor the vehicle's surroundings, traffic conditions, or infrastructure status. The data from the agent sensor is transmitted to the vehicle's control system, enabling real-time adjustments to navigation, safety protocols, or performance parameters. The integration of off-board sensors improves the vehicle's ability to detect hazards, optimize routes, and respond to dynamic conditions, enhancing overall safety and efficiency. The system may also include processing units that analyze the sensor data to generate actionable insights or alerts for the vehicle operator or autonomous control systems. This approach leverages external sensing capabilities to overcome the limitations of on-board sensors, particularly in complex or high-risk environments.
17. The system of claim 10 , wherein to identify the relevant sensor comprises to: identify an infrastructure sensor that is within a geofence of the AV.
Autonomous vehicles (AVs) rely on sensor data to navigate and make decisions, but integrating data from multiple sources, including infrastructure sensors, presents challenges in identifying relevant data sources. This invention addresses the problem of efficiently determining which infrastructure sensors are relevant to an autonomous vehicle's current operations. The system identifies infrastructure sensors that are within a predefined geofence around the AV, ensuring that only the most pertinent sensor data is processed. The geofence is dynamically adjusted based on the AV's location and operational needs, such as avoiding obstacles or optimizing route planning. By filtering sensor data based on proximity, the system reduces computational overhead and improves real-time decision-making. The invention also includes mechanisms to prioritize sensor data based on relevance, ensuring critical information is processed first. This approach enhances the AV's situational awareness while minimizing unnecessary data processing. The system may also integrate with other AV subsystems, such as perception modules, to further refine sensor selection and improve overall autonomy.
18. The system of claim 10 , wherein to identify the relevant sensor further comprises to: analyze first contents of a first image from an on-board sensor; analyze second contents of a second image from an off-board sensor; and in response to comparing the first contents to the second contents, identify the off-board sensor to the tele-operator.
This invention relates to a system for identifying relevant sensors in a tele-operated environment, particularly for remote operation of vehicles or machinery. The system addresses the challenge of determining which sensors provide the most useful data to a tele-operator, ensuring efficient and accurate remote control. The system includes multiple sensors, both on-board (mounted on the vehicle or machinery) and off-board (external to the vehicle or machinery), which capture images or other data. The system analyzes the contents of images from these sensors to determine their relevance. Specifically, it compares the contents of a first image from an on-board sensor with the contents of a second image from an off-board sensor. Based on this comparison, the system identifies the most relevant sensor to the tele-operator, ensuring that the operator receives the most useful data for effective remote operation. This comparison may involve matching features, detecting objects, or assessing environmental conditions to determine which sensor provides the best perspective or information. The system dynamically selects the appropriate sensor to enhance situational awareness and decision-making for the tele-operator.
19. A system for exception handing in autonomous driving, comprising: a memory; and a processor, the processor is configured to execute instructions stored in the memory to: receive an assistance request to resolve an exception situation from an autonomous vehicle (AV); identify an on-board sensor of the AV; identify an off-board sensor; select one of the on-board sensor or the off-board sensor as a selected sensor; present data from the selected sensor to a tele-operator; present tools to the tele-operator, wherein the tools comprise a path-drawing tool, a first action usable by the tele-operator to accept a proposed trajectory received from the AV, and a second action usable by the tele-operator to reject the proposed trajectory; receive, from the tele-operator, a validated solution to the exception situation; and transmit the validated solution to the AV.
The system addresses exception handling in autonomous driving by enabling remote tele-operators to assist autonomous vehicles (AVs) in resolving unexpected situations. The system includes a memory and a processor that executes instructions to receive an assistance request from an AV encountering an exception situation. The processor identifies both on-board sensors (e.g., cameras, LiDAR) and off-board sensors (e.g., traffic cameras, roadside sensors) available to the AV. It then selects one of these sensors to provide the tele-operator with relevant data for assessing the situation. The system presents this sensor data to the tele-operator along with tools to assist in decision-making. These tools include a path-drawing tool for manually defining a trajectory, an action to accept a proposed trajectory generated by the AV, and another action to reject the proposed trajectory. The tele-operator uses these tools to validate a solution, which is then transmitted back to the AV for execution. This approach ensures human oversight in critical situations where the AV may require external guidance.
20. The system of claim 19 , wherein to select one of the on-board sensor or the off-board sensor comprises to: in response to determining that an image from the on-board sensor is saturated with light, select the off-board sensor.
This invention relates to a sensor selection system for autonomous vehicles or robotic systems, addressing the challenge of unreliable sensor data due to environmental conditions like excessive light saturation. The system dynamically selects between on-board and off-board sensors to ensure accurate perception and navigation. On-board sensors are typically mounted on the vehicle or robot, while off-board sensors are external, such as cloud-based or infrastructure-mounted cameras. The system monitors the quality of on-board sensor data, particularly image saturation from excessive light. When an on-board image is determined to be saturated, the system automatically switches to the off-board sensor to maintain reliable data input. This selection process ensures continuous operation even in adverse lighting conditions, improving safety and functionality. The system may also include additional sensor fusion techniques to integrate data from multiple sources, enhancing overall system robustness. The invention is particularly useful in autonomous driving, where environmental factors can degrade sensor performance, and reliable perception is critical for safe operation.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
July 18, 2019
February 15, 2022
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.