Early object detection for unprotected turns

This application is a continuation of U.S. patent application Ser. No. 17/728,375, filed Apr. 25, 2022, which is a divisional of U.S. patent application Ser. No. 16/679,723, filed Nov. 11, 2019 and issued as U.S. Pat. No. 11,453,392 on Sep. 27, 2022, which is a continuation of U.S. patent application Ser. No. 15/834,535, filed Dec. 7, 2017, and issued as U.S. Pat. No. 10,501,085 on Dec. 10, 2019, the entire disclosures of which are incorporated herein by reference.

Autonomous vehicles, for instance, vehicles that do not require a human driver, can be used to aid in the transport of passengers or items from one location to another. Such vehicles may operate in a fully autonomous mode where passengers may provide some initial input, such as a pickup or destination location, and the vehicle maneuvers itself to that location.

While some autonomous vehicles are operable in a semi-autonomous mode, such as where a human driver takes over control of the vehicle in some circumstances, it is nevertheless important for the autonomous vehicle to operate in the safest manner possible.

One aspect of the disclosure provides a method for maneuvering a vehicle. The method includes defining, for an autonomous vehicle having one or more sensors, a first sensor range having a first associated confidence value, the first sensor range adjacent an outer bound of reach of the one or more sensors, and defining a second sensor range, the second sensor range having a second associated confidence value that is higher than the first confidence value, the second sensor range being within the first sensor range and closer to the one or more sensors than the first sensor range. The method further includes receiving, at one or more processors, input from the one or more sensors, detecting, by the one or more processors based on the input received from the one or more sensors, an object within the first sensor range of the vehicle, and causing the vehicle to take a first action in response to detecting the object within the first sensor range, prior to the object being detected within the second sensor range, wherein the first action comprises at least one of yielding, ceasing acceleration, decelerating, or switching lanes. In some examples, the method may further include detecting, by the one or more processors based on further input received from the one or more sensors, that the object has moved into the second sensor range of the vehicle, and validating the detection of the object within the first sensor range based on the detection of the object within the second sensor range.

The vehicle, when receiving the input from the one or more sensors, may be waiting to merge onto a roadway having a speed limit of 45 miles per hour or higher. In this example, the object detected within the first sensor range may be a second vehicle traveling on the roadway, and the first action may include continuing to wait to merge onto the roadway. In another example, the vehicle, when receiving the input from the one or more sensors, is traveling in a first lane of a roadway having a speed limit of 45 miles per hour or higher, the object detected within the first sensor range is a second vehicle traveling in the first or a second lane on the roadway, and approaching the vehicle from behind, and the first action includes changing lanes away from the lane in which the second vehicle is traveling. In yet a further example, the vehicle, when receiving the input from the one or more sensors, is making an unprotected turn onto a roadway, the object detected within the first sensor range is a second vehicle traveling along the roadway towards the vehicle, and the first action may include waiting to make the unprotected turn until the first and second sensor ranges are clear.

Another aspect of the disclosure provides a system, including one or more sensors, the one or more sensors having a first defined range adjacent an outer bound of reach of the one or more sensors, the first defined range having a first associated confidence value, and a second defined range within the first sensor range and closer to the one or more sensors than the first sensor range, the second defined range having a second associated confidence value that is higher than the first confidence value. The system further includes one or more processors in communication with the one or more sensors, wherein the one or more processors are configured to receive input from the one or more sensors, detect, based on the input received from the one or more sensors, an object within the first sensor range of the vehicle, and cause the vehicle to take a first action in response to detecting the object within the first sensor range, prior to the object being detected within the second sensor range, wherein the first action comprises at least one of yielding, ceasing acceleration, decelerating, or switching lanes.

Yet another aspect of the disclosure provides a vehicle, including a driving system including at least an acceleration system, a braking system, and a steering system, and a perception system including at least one or more sensors. The one or more sensors have a first defined range adjacent an outer bound of reach of the one or more sensors, the first defined range having a first associated confidence value, and a second defined range within the first sensor range and closer to the one or more sensors than the first sensor range, the second defined range having a second associated confidence value that is higher than the first confidence value. The vehicle further includes a control system including at least one or more processors in communication with the one or more sensors. The one or more processors are configured to receive input from the one or more sensors, detect, based on the input received from the one or more sensors, an object within the first sensor range of the vehicle, and cause the vehicle to take a first action in response to detecting the object within the first sensor range, prior to the object being detected within the second sensor range, wherein the first action comprises at least one of yielding, ceasing acceleration, decelerating, or switching lanes. The vehicle of may in some examples be autonomous.

The technology relates to detection of objects by an autonomous vehicle. In particular, in some driving situations, the autonomous vehicle may operate based on an early detection of an object, without waiting for a more precise detection. For example, whereas an autonomous vehicle may typically wait until objects are detected within a sensor range having a relatively high associated confidence value, the present technology provides for earlier detection based on detection within a noisier sensor range.

An autonomous vehicle according to the present technology has a multi-level sensor range, wherein a first, less precise level of the sensor range is adjacent an outer bounds of the sensor range, and a second, more precise level of the sensor range is inside the first range. As such, the first range may have a lower associated confidence value than the second range. For example, objects detected within the first range are likely to actually be present with a first degree of certainty, while objects detected within the second range are likely to actually be present with a second degree of certainty which is higher than the first.

In some cases, an object (e.g., house, foliage, parked car, etc.) may be blocking a sensor's view. In such cases, the “first range” may be immediately adjacent to the occluded area, where a moving object would first emerge from occlusion but would still have a lower confidence value. The “second range” may be adjacent to the first range, but further downstream from the occluded area, where confidence values are higher.

Driving situations in which early detection may be used may include those in which the autonomous vehicle is expected to yield to other traffic, such as when merging onto a highway, or when crossing one or more lanes of traffic to make a left turn. Such early detection is particularly advantageous when oncoming traffic is traveling at a high rate of speed, such as 50 mph or above. In such instances, it may take too long for the autonomous vehicle to receive higher quality detection in time to take the required driving action safely, such as merging or turning. By responding to noisier detections, or objects perceived with a lower degree of confidence, an appropriate driving action can safely be taken earlier, and thus within time before the detected object becomes too close. The appropriate driving action may be, for example, waiting to merge or turn until no oncoming objects are detected within a field of view of the sensors.

Detecting objects within the first sensor range, having the lower confidence value, may produce false positives. However, responding as if an object is present, when it actually is not, is typically a safer response.

The sensors used to detect objects may include any of a variety of sensors, including those typically included in autonomous vehicles. For example, the sensors may include radar, LIDAR, sonar, or cameras.

In some examples, a confidence level of objects detected within the first sensor range may be improved using various techniques. For example, input received from sensors within the first sensor range may be filtered, such that only detections at or above a threshold signal-to-noise ratio (SNR) will trigger a reaction. This reaction may be different than a reaction to a detection confirmed in the second sensor range. For example, a preliminary reaction triggered by detection within the first sensor range may include limited braking, whereas a confirmed detection in the second sensor range may trigger full braking. In other examples, different types of sensors may be used to detect the object within the first sensor range, and the input from those different types of sensors may be cross-validated. For example, if both a radar sensor and a LIDAR sensor detected the object within the first sensor range, the object may be determined to be present. In contrast, if only the radar sensor detected the object within the first sensor range, and the LIDAR did not, the object may be determined to not actually be present.

In other examples, a confidence level of objects detected across a pipeline of both the first sensor range and the second sensor range may also be improved. For example, in addition to extracting features from a current set of perception data, such as SNR, cross-validation, etc., the object may be tracked temporally over multiple iterations of perception data. For example, object properties having temporal elements, such as speed and acceleration, may be tracked.

As shown in, a vehiclein accordance with one aspect of the disclosure includes various components. While certain aspects of the disclosure are particularly useful in connection with specific types of vehicles, the vehicle may be any type of vehicle including, but not limited to, cars, trucks, motorcycles, buses, recreational vehicles, etc. The vehicle may have one or more computing devices, such as computing devicescontaining one or more processors, memoryand other components typically present in general purpose computing devices.

The memorystores information accessible by the one or more processors, including instructionsand datathat may be executed or otherwise used by the processor. The memorymay be of any type capable of storing information accessible by the processor, including a computing device-readable medium, or other medium that stores data that may be read with the aid of an electronic device, such as a hard-drive, memory card, ROM, RAM, DVD or other optical disks, as well as other write-capable and read-only memories. Systems and methods may include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media.

The instructionsmay be any set of instructions to be executed directly (such as machine code) or indirectly (such as scripts) by the processor. For example, the instructions may be stored as computing device code on the computing device-readable medium. In that regard, the terms “instructions” and “programs” may be used interchangeably herein. The instructions may be stored in object code format for direct processing by the processor, or in any other computing device language including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. Functions, methods and routines of the instructions are explained in more detail below.

The datamay be retrieved, stored or modified by processorin accordance with the instructions. For instance, although the claimed subject matter is not limited by any particular data structure, the data may be stored in computing device registers, in a relational database as a table having a plurality of different fields and records, XML documents or flat files. The data may also be formatted in any computing device-readable format.

The one or more processorsmay be any conventional processors, such as commercially available CPUs. Alternatively, the one or more processors may be a dedicated device such as an ASIC or other hardware-based processor. Althoughfunctionally illustrates the processor, memory, and other elements of computing devicesas being within the same block, it will be understood by those of ordinary skill in the art that the processor, computing device, or memory may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. For example, memory may be a hard drive or other storage media located in a housing different from that of computing devices. Accordingly, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

Computing devicesmay include all of the components normally used in connection with a computing device such as the processor and memory described above as well as a user input(e.g., a mouse, keyboard, touch screen and/or microphone) and various electronic displays (e.g., a monitor having a screen or any other electrical device that is operable to display information). In this example, the vehicle includes an internal electronic displayas well as one or more speakersto provide information or audio visual experiences. In this regard, internal electronic displaymay be located within a cabin of vehicleand may be used by computing devicesto provide information to passengers within the vehicle.

Computing devicesmay also include one or more wireless network connectionsto facilitate communication with other computing devices, such as the client computing devices and server computing devices described in detail below. The wireless network connections may include short range communication protocols such as Bluetooth, Bluetooth low energy (LE), cellular connections, as well as various configurations and protocols including the Internet, World Wide Web, intranets, virtual private networks, wide area networks, local networks, private networks using communication protocols proprietary to one or more companies, Ethernet, WiFi and HTTP, and various combinations of the foregoing.

In one example, computing devicesmay be control computing devices of an autonomous driving computing system or incorporated into vehicle. The autonomous driving computing system may capable of communicating with various components of the vehicle in order to control the movement of vehicleaccording to primary vehicle control code of memory. For example, returning to, computing devicesmay be in communication with various systems of vehicle, such as deceleration system, acceleration system, steering system, signaling system, navigation system, positioning system, perception system, and power system(i.e. the vehicle's engine or motor) in order to control the movement, speed, etc. of vehiclein accordance with the instructionsof memory. Again, although these systems are shown as external to computing devices, in actuality, these systems may also be incorporated into computing devices, again as an autonomous driving computing system for controlling vehicle.

As an example, computing devicesmay interact with one or more actuators of the deceleration systemand/or acceleration system, such as brakes, accelerator pedal, and/or the engine or motor of the vehicle, in order to control the speed of the vehicle. Similarly, one or more actuators of the steering system, such as a steering wheel, steering shaft, and/or pinion and rack in a rack and pinion system, may be used by computing devicesin order to control the direction of vehicle. For example, if vehicleis configured for use on a road, such as a car or truck, the steering system may include one or more actuators to control the angle of wheels to turn the vehicle. Signaling systemmay be used by computing devicesin order to signal the vehicle's intent to other drivers or vehicles, for example, by lighting turn signals or brake lights when needed.

Navigation systemmay be used by computing devicesin order to determine and follow a route to a location. In this regard, the navigation systemand/or datamay store detailed map information, e.g., highly detailed maps identifying the shape and elevation of roadways, lane lines, intersections, crosswalks, speed limits, traffic signals, buildings, signs, real time traffic information, vegetation, or other such objects and information.

Positioning systemmay be used by computing devicesin order to determine the vehicle's relative or absolute position on a map or on the earth. For example, the position systemmay include a GPS receiver to determine the device's latitude, longitude and/or altitude position. Other location systems such as laser-based localization systems, inertial-aided GPS, or camera-based localization may also be used to identify the location of the vehicle. The location of the vehicle may include an absolute geographical location, such as latitude, longitude, and altitude as well as relative location information, such as location relative to other cars immediately around it which can often be determined with less noise than absolute geographical location.

The positioning systemmay also include other devices in communication with computing devices, such as an accelerometer, gyroscope or another direction/speed detection device to determine the direction and speed of the vehicle or changes thereto. By way of example only, an acceleration device may determine its pitch, yaw or roll (or changes thereto) relative to the direction of gravity or a plane perpendicular thereto. The device may also track increases or decreases in speed and the direction of such changes. The device's provision of location and orientation data as set forth herein may be provided automatically to the computing devices, other computing devices and combinations of the foregoing.

The perception systemalso includes one or more components for detecting objects external to the vehicle such as other vehicles, obstacles in the roadway, traffic signals, signs, trees, etc. For example, the perception systemmay include lasers, sonar, radar, cameras and/or any other detection devices that record data which may be processed by computing device. In the case where the vehicle is a passenger vehicle such as a minivan, the minivan may include a laser or other sensors mounted on the roof or other convenient location. For instance,is an example external view of vehicle. In this example, roof-top housingand dome housingmay include a lidar sensor as well as various cameras and radar units. In addition, housinglocated at the front end of vehicleand housings,on the driver's and passenger's sides of the vehicle may each store a lidar sensor. For example, housingis located in front of driver door. Vehiclealso includes housings,for radar units and/or cameras also located on the roof of vehicle. Additional radar units and cameras (not shown) may be located at the front and rear ends of vehicleand/or on other positions along the roof or roof-top housing.

The computing devicesmay control the direction and speed of the vehicle by controlling various components. By way of example, computing devicesmay navigate the vehicle to a destination location completely autonomously using data from the detailed map information and navigation system. Computing devicesmay use the positioning systemto determine the vehicle's location and perception systemto detect and respond to objects when needed to reach the location safely. In order to do so, computing devicesmay cause the vehicle to accelerate (e.g., by increasing fuel or other energy provided to the engine by acceleration system), decelerate (e.g., by decreasing the fuel supplied to the engine, changing gears, and/or by applying brakes by deceleration system), change direction (e.g., by turning the front or rear wheels of vehicleby steering system), and signal such changes (e.g., by lighting turn signals of signaling system). Thus, the acceleration systemand deceleration systemmay be a part of a drivetrain that includes various components between an engine of the vehicle and the wheels of the vehicle. Again, by controlling these systems, computing devicesmay also control the drivetrain of the vehicle in order to maneuver the vehicle autonomously.

illustrates an example of defined sensor ranges, wherein different vehicle responses may be triggered by detection of objects in the different sensor ranges. As mentioned above, the vehiclehas one or more sensors, such as radar, LIDAR, cameras, etc. The one or more sensors have an outer bound of reach, illustrated by boundary. For example, the boundarymay represent an extent of a field of view of the one or more sensors, such that the sensors do not detect objects outside the boundary. In some examples, the sensor field of view may be occluded, such as by a tree, building, etc. In this case, the outer boundarymay be at least partially defined by the occluding object. In other examples, a distance between the sensor on the vehicle and the outer boundarymay depend on, for example, the type of sensor used. For example, a radar sensor may have a reach of approximately 160-200 m, while a LIDAR sensor may have a reach of approximately 85-160 m.

Within the outer boundaryis a first sensor range. The first sensor range has a first confidence value associated therewith. Sensor signals reaching the first range and near the outer boundarymay be noisy because of long range and/or short distance traversed inside a reach of the sensor (e.g., less time within a sensor field of view). Accordingly, the first confidence value may be relatively low.

Within the first range sensor range, and closer to the vehicle, is second sensor range. Boundarymay define the first sensor rangefrom the second sensor range. The second sensor rangemay be, for example, approximately 10-30 m shorter in range than the first sensor range, or more. The second sensor rangeis associated with a second confidence value. Signals within the second sensor rangeare less noisy and more precise than sensor signals within the first sensor range, because the signals in the second sensor rangeare not as far reaching as those in the first sensor range. Accordingly, the second confidence value is higher than the first confidence value.

According to some examples, the boundarybetween the first sensor rangeand the second sensor rangemay be determined based on signal quality, distance from or bearing to the one or more sensors, and/or other factors. By way of example only, the boundarymay be set at a threshold distance, a threshold signal to noise ratio, or some combination thereof. In this regard, the boundarymay be varied along with changes in signal quality. In other examples, the boundarymay be redefined to accommodate for other factors, such as weather, visibility, sensor degradation, etc.

While only two sensor ranges are defined in the example described above, it should be understood that additional sensor ranges may also be defined, such as a third sensor range, fourth sensor range, etc.

Different actions of the vehiclemay be triggered based on the sensor range in which an object is detected. For example, if an object is detected in the first sensor range, the vehiclemay take a preliminary action, such as yielding, decelerating, ceasing acceleration, changing lanes, waiting, etc. If the object moves within the second sensor range, however, the vehiclemay take a different responsive action, such as stopping, or another action. In some examples, detection of the object within the second rangemay serve as a verification of the previous detection of the object within the first range. In this regard, the first action taken in response to the first detection may be considered a preliminary action in preparation for the second action taken in response to the second detection.

In addition to the operations described above and illustrated in the figures, various operations will now be described. It should be understood that the following operations do not have to be performed in the precise order described below. Rather, various steps can be handled in a different order or simultaneously, and steps may also be added or omitted.

is a flow diagram illustrating an example method of designating multiple ranges of sensor field of view, and operating an autonomous vehicle in response to detection of objects in one of the multiple ranges.

In block, a first sensor range is defined at an outer bounds of a field of view of one or more sensors on a vehicle. The first sensor range has a first associated confidence value. The first confidence value may be determined based on, for example, signal quality, error rate, or other information. Because the first sensor range is near an outer bounds of the field of view, input signals received at the one or more sensors are likely to be noisier than other portions of the field of view. Accordingly, the confidence value for the first sensor range may be relatively low as compared to other portions of the field of view.

In block, a second sensor range is defined, wherein the second sensor range is closer to the one or more sensors than first sensor range, such as shown in. The second sensor range has a second associated confidence value, wherein the second confidence value is higher than the first confidence value. For example, signals within the second sensor range, which travel less distance relative to signals traveling to the first sensor range, may be more accurate and less noisy. Additionally, objects that have come within the second range have traveled a greater distance within the overall detectable range. A boundary between the first sensor range and the second sensor range may be determined, for example, based on the associated confidence values or other accuracy quantifiers. For example, the boundary may be set based on a threshold distance, a threshold confidence value, or the like.

In block, one or more processors of the vehicle receive input from the one or more sensors. According to some examples, where the one or more sensors include a plurality of different types of sensors, the received input may include input from multiple sources.

In block, an object within the first sensor range is detected based on the received input. The object may be, for example, a pedestrian, a cyclist, another vehicle, or any other object the vehicle should avoid. In many examples, the object may be moving or about to move. For example, the object may be approaching the vehicle on the driver side as the vehicle prepares to make a left turn. In other examples, the object may be approaching the vehicle from behind as the vehicle travels on a roadway. In other examples, the object may be traveling in a right lane of a highway onto which the vehicle is waiting to merge. While these are merely a few examples, it should be understood that the object may be detected in any of a number of circumstances. In the example where multiple sensors of different types are providing input to one or more processors, the object may be detected by two or more of the different types of sensors. The signals from the different types of sensors may thus be used to cross-validate one another.

In block, in response to detecting the object within the first sensor range, the vehicle is triggered to take a first action prior to the object being detected within the second sensor range. The first action may be, for example, waiting to turn or merge, decelerating, ceasing acceleration, or another precautionary action.

In some examples, taking the action in response to detecting the object within the first sensor range may be limited based on further criteria. For example, the action may only be taken if the signal to noise ratio of signals detecting the object is at or above a predetermined threshold, such as by filtering out signals with a signal to noise ratio below the predetermined threshold. Where different types of sensors are used to cross-validate one another, the action may only be taken if the input is cross-validated.

In some examples, the method may further include detecting, based on further input received from the one or more sensors, that the object has moved into the second sensor range of the vehicle. In such examples the detection of the object within the second range may be used to validate the detection of the object in the first range. Accordingly, the vehicle may take a secondary action in response, such as stopping, continuing to wait, etc.

illustrate some example scenarios in which the vehicle responds to detection of objects within the first sensor range. As shown in, the vehicleis traveling on an entry rampto roadway. The roadwaymay be a high speed roadway, such as where vehicles travel approximately 40 miles per hour and faster. The second sensor range, which has a perimeter closer to the vehiclethan the first sensor range, is clear. In other words, no objects are detected within the second sensor range. If the vehiclewere to rely solely on this indication from the second sensor range, it may proceed to enter the roadwayand either “cut off” the vehicleor cause a collision or near-collision with the vehicle. The first sensor range, however, includes the vehicle. Accordingly, although the first sensor rangemay have a lower confidence value, its indication of oncoming vehicle may be heeded as precaution. As such, the vehiclemay yield to the vehicle, and wait to merge onto the roadway. If detection of the vehiclein the first sensor rangeis a false positive, and no vehicle is actually present within the first sensor range, yielding or waiting should not cause any danger due to the limited nature of reactions triggered by detections within the first sensor range.

As shown in, the vehiclehas moved into the second sensor range. Such detection of the vehiclewithin the second sensor rangemay be used to validate the detection in the first sensor range. According to some examples, such validation may be used for adjusting confidence values for future detections in the first sensor range, calibrating sensors, training the one or more processors to trigger reactions for detections within the first sensor range, or for any other purpose. For example, if at a future time in a similar circumstance the vehicledetects an object resembling a vehicle within the first sensor range, the one or more processors of the vehiclemay determine with greater confidence that a vehicle is actually approaching.

As shown in the example of, while vehicleis traveling on the roadwaytowards the entrance ramp, it is so far away from the vehiclethat it is outside the field of view. In this example, the maximum sensor range of the vehicleis sufficiently long that a clear field of view would indicate a sufficient buffer between the vehicleand the vehiclefor the vehicleto safely enter roadway. The vehicleis beyond the outer bounds of the first sensor range, and thus beyond a point of detection by any of the sensors on the vehicle. Because the vehicleis outside the field of view, it may be far enough away from the vehiclethat the vehiclecan safely enter the roadway. Accordingly, as both the first and second sensor ranges are clear and sufficiently large to safely detect traffic, and no oncoming traffic is detected at even a low signal to noise ratio, the vehicleproceeds to merge onto the roadway.

illustrates another scenario, where the vehicleis planning to make an unprotected left turn onto roadwayalong path. An unprotected turn in this example is one where the vehiclecrosses a lane of traffic that has the right of way. While the vehiclemust stop at stop sign, there are no signs, traffic lights, or other requirements for traffic traveling along the roadwayto stop, slow down, yield, or the like. Thus, as the vehiclemakes a left turn through intersectionand into roadway, there is some risk that a vehicle traveling towards the intersection, such as the vehicle, will collide with the vehicleif the vehicleenters the intersection. Accordingly, the vehicledetects whether any vehicles are approaching to avoid such a collision. To give the vehicleas much notice as possible, the vehicledetects whether any objects are within the first sensor range. Accordingly, the vehiclemay detect at least a portion of the vehicle. In response, the vehiclemay continue to wait at the stop signuntil the field of view is clear.

Within the first sensor range, the vehiclealso detects pedestrianwho has begun crossing the roadway. Accordingly, in this context the vehiclemay also respond to the pedestrianby waiting to make the turn.

While several examples of maneuvers by the vehiclebenefitting from early perception in the first sensor range are described above, it should be understood that these examples are in no way limiting and that the technology may be applied in any of a number of other instances. For example, where traffic in an adjacent lane on a high speed roadway is moving faster than the vehicle, the vehicle may benefit by early detection of such traffic. In response, the vehicle may change lanes or take another precautionary action. Other examples may include any situation in which the vehicle should yield to another vehicle which it cannot yet see with high confidence.