Dynamic Parking Location Determining Management

Machine learned models can be employed to predict an action for a variety of robotic devices. For instance, planning systems in autonomous and semi-autonomous vehicles determine actions for a vehicle to take in an operating environment. Actions for a vehicle may be determined based in part on avoiding objects present in the environment. For example, an action may be generated to yield to a pedestrian, to change a lane to avoid another vehicle in the road, or the like. Accurately predicting future object trajectories (and drivable areas in an environment where a trajectory may traverse) may be necessary to safely operate the vehicle in the vicinity of the objects.

This application describes techniques for determining a dynamic drivable area. Drivable areas can be determined by machine learned models. The machine learned models can generate outputs including probabilities that elements of the outputs represent the drivable areas and non-drivable areas in environments. In some examples, the machine learned models can generate outputs including probabilities that elements of the outputs represent parking locations or drivable areas in an environment. The outputs can be utilized to generate trajectories, which can in turn be utilized to control vehicles traversing the environments. Training of the machine learned models can be based on sensor data and map data. The sensor data and the map data can be utilized to determine representations of the environments, such as top-down representations. The top-down representations can include at least road marking and velocity information associated with dynamic objects in the environments. The sensor data can be utilized to determine the dynamic objects.

The vehicles can be controlled based on various types of areas. The areas can include construction zones, parking destination map areas, and/or other incident areas. The construction zones can be identified based on various types of construction objects. The construction objects can include dividers, cones, and so on, or any combination thereof. The parking area destination map areas can include curbs, lane markings, and so on, or any combination thereof. Examples of other incident areas include, but are not limited to, areas defined by police tape or caution tape, areas associated with an emergency (e.g., ambulance, fire, police, etc.), areas defined by temporary barriers (e.g., protests, parades, bike/foot races, etc.), and the like.

The top-down representation can include a multi-channel image or polylines, which can be input into a machine learned model. The machine learned model can analyze the top-down representation to generate outputs utilized to identify drivable or non-drivable areas. The multi-channel image can include rasterized image data. For example, the rasterized image data can include pixels and/or picture elements that include unique colors and tonal characteristics that are combinable together to create an image, including the top-down representation. In some examples, different semantic data can be presented in different channels or layers of the multi-channel image. The polylines can be as vectorized image data. The polylines can include scalable vector graphics (SVG) shapes that create lines (straight or otherwise) connecting several points.

The machine learned models can be trained based on vehicle data captured as the vehicles traverse the environments. Data associated with the environments can be collected based on input to sensors of the vehicles. The data can include perception data that is utilized to identify previous road marking and velocity information. The previous road marking and velocity information can include velocity information associated with objects in the environment. The velocity information can include velocity data, such as a speed, a direction, etc., associated with the objects. The perception data can be identified based on sensor data collected by the vehicles. The sensor data can include data that is more current and/or accurate than map data received by the vehicles. For example, the sensor data can include actual data indicating and/or representing locations, characteristics, identities, etc., or any combination thereof, of objects in the construction zones and parking destination map areas.

The dynamic drivable area determining management techniques described herein can improve a functioning of a computing device by providing drivable area and non-drivable area data for performing operations to control an autonomous vehicle (or other system). For example, drivable areas and non-drivable areas can be identified based on dynamically generated data. The dynamically determined drivable areas and non-drivable areas can be identified and utilized as part of subsequent processes such as localization, perception (e.g., detecting, identifying, segmenting, classifying, tracking, etc.), route planning, trajectory generation, and the like. By utilizing the dynamic data, the processes can be performed more accurately, with consumption of less processing power, and/or with storage in relatively smaller amounts of memory.

For example, in some instances, faster and/or more accurate drivable area determination can be used in generating a trajectory of an autonomous vehicle, which can improve safety for occupants of an autonomous vehicle. Further, in some examples, the techniques discussed herein can be used to verify a calibration of sensors, can provide error checking or voting to determine if a sensor measurement is inaccurate (e.g., by comparing a depth measurement to another depth sensor), and/or can be used as a fallback in the event other sensors are occluded or disabled. In some examples, training a machine learned model using such self-supervised and supervised techniques (which together may comprise semi-supervised training) provide for a machine learned model that may output more accurate depth estimates than a model trained without these techniques. These and other improvements to the functioning of the computer are discussed herein.

The techniques described herein can be implemented in a number of ways. Example implementations are provided below with reference to the following figures. Although applicable to vehicles, such as autonomous vehicles, the methods, apparatuses, and systems described herein can be applied to a variety of systems and are not limited to autonomous vehicles. In one example, similar techniques may be utilized in driver controlled vehicles in which such a system may provide an indication of whether it is safe to perform various maneuvers. In another example, the techniques can be utilized in an aviation or nautical context, or in any system configure to input data to determine movement associated with objects in an environment. Additionally, the techniques described herein can be used with real data (e.g., captured using sensor(s)), simulated data (e.g., generated by a simulator), or any third of the two.

are flowcharts depicting example processes for dynamic drivable area determining management. With respect to, a processis depicted in a flowchart for dynamic drivable area determining management. At an operation, the processcan include determining a dynamic object. A vehicle (e.g., the vehicle, as discussed below in further detail) receive sensor data and can process the sensor data to determine that a dynamic object is present in the environment. In some examples, the operationcan include determining dynamic drivable area determining information (or “dynamic information”) associated with the environment. The dynamic information, for example, may be determined based on the dynamic object. In some examples, a dynamic object may represent one or more dynamic objects.

The dynamic information may include information associated with any portions of the environment being dynamically and/or continually updated. The dynamic information may include any of the vehicles, moving objects, stationary objects, and/or road markings in the environment.

In various examples, a dynamic object may be determined based on sensor data. Determining the dynamic object may include determining the sensor data, and the dynamic object(s) based on the sensor data. The sensor data may include image data (e.g., camera data) and/or depth data (e.g., lidar data). In some examples, the sensor data may include data generated by one or more image sensors (e.g., one or more red-green-blue (RGB) cameras, one or more intensity cameras (greyscale), one or more infrared cameras, one or more ultraviolet cameras, and the like), one or more depth cameras (e.g., RGB D cameras), one or more time-of-flight (ToF) sensors, one or more lidar sensors, one or more radar sensors, one or more sonar sensors, and the like.

An exampleillustrates a portion of an environment including one or more vehicles, such as a vehicle. The environment illustrated in examplemay include one or more dynamic objects, such as a dynamic object. The environment may include one or more cones, as represented by circles illustrated in.

Individual ones of the vehicle(s) may include an autonomous vehicle. Individual ones of the dynamic objects may include a non-autonomous vehicle. However, the disclosure is not limited as such, and individual ones of the vehicle(s) and/or the dynamic objects may include an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

The vehiclecan be controlled to follow a dynamic object (e.g., the dynamic object, as discussed below in further detail,) based on the moving velocity of the dynamic objectindicating that the dynamic objectis following a safe path. Characteristics, such as the velocity, of the dynamic objectmay be utilized to identify a safe path associated with the dynamic object. The safe path may be identified since the dynamic objectis likely to proceed at a velocity above a threshold velocity only if the path of the dynamic objectis safe.

Various characteristics associated with the dynamic objectcan be utilized to identify the safe path. For example, one or more characteristics, such as one or more locations, one or more velocities, one or more directionalities of the dynamic objectat one or more times, respectively, in the environment, can be utilized to control the vehicle. The characteristic(s) may inform a machine learned (ML) model (e.g., the machine learned (ML) model, as discussed below) that areas that otherwise would be non-drivable are actually drivable areas.

An operation, the processcan include determining a top-down representation. The top-down representation can include any portion of the environment. For example, the top-down representation can include one or more zones (or “zone(s)”) (or “area(s)”). The zone(s) may include one or more construction zones. Determining the top-down representation can include determining one or more other zones. A current and/or future location of the dynamic object(s) and/or the vehicle(s) may be in, and/or near, the zone(s).

One or more areas of the environment may be identified as one or more initial drivable areas. The initial drivable area(s) may be identified in the area(s), such as the construction zone(s). The initial drivable area(s) may be identified based on map data and sensor data.

In some examples, the top-down representation may represent one or more top-down representations. In those or other examples, determining the top-down representation can include determining the top-down representation(s), which may include any portion of the environment.

An exampleillustrates the environment, with the top-down representation(s) that include various portions of the environment. The top-down representation(s) may include the vehicle, the dynamic object, the zone(s), and/or the area(s). For example, the vehicle can determine, as part of the dynamic information, the top-down representation(s), which can include a top-down representationwith the vehicleand the dynamic object. In such an example or another example, the zone(s) in the top-down representationmay include a construction zone.

The top-down representationmay include an area (e.g., an initial drivable area)being initially identified as being drivable. The initial drivable areamay overlap (e.g., partially or entirely overlap) with the construction zone. The initial drivable areamay be initially identified, due to data (e.g., the map data and/or the sensor data) being out of date, as an area within the top-down representationthat is subsequently identified as being not drivable.

The top-down representationmay include different channels, such as different snapshots (e.g., dimensions) associated with different times (e.g., points in time) (e.g., temporal dimensions). The channels may be included in a multi-channel image of multi-channel image data with rasterized input (e.g., the rasterized input, as discussed above with reference to). For example, the temporal dimensions may be represented by a group of rectangles identified by reference numeral, as illustrated in.

The dynamic objectmay move in, and/or near, the zone(s). The dynamic objectmay move around the construction zone. The dynamic objectmay have a velocity that is above a threshold velocity.

At operation, the processcan include inputting the top-down representation (or “representation”) into a machine learned (ML) model. Inputting the representation can include inputting, into the ML model, the top-down representationincluding the vehicle, the dynamic object, and the construction zone.

An exampleillustrates the environment, with the top-down representation(s) being input into the machine learned (ML) model. The machine learned (ML) model may represent one or more machine learned (ML) models, such as a machine learned model (ML). The top-down representation(s) being input into the machine learned model (ML)can include the top-down representationwith the vehicle, the dynamic object, and the construction zone.

At operation, the processcan include receiving outputs with probabilities. The outputs can include an output with a probability that an element of the output represents a drivable area. The output can include a first output; and the probability can include a first probability that an element of the first output represents the drivable area. The outputs can include an output with a probability that an element of the output represents a non-drivable area. The output can include a second output; and the probability can include a second probability that an element of the second output represents the non-drivable area.

An exampleillustrates the environment, with one or more probabilities output by the machine learned (ML) model. The vehiclecan determine one or more outputs with the probability(ies) that one or more elements of the output(s) represent one or more drivable areas and/or one or more non-drivable areas. The output(s) may be included in the dynamic information.

The probability(ies) can include a probability P, e.g., represented by a star encircling a portion (e.g., a pixel) of the top-down representation (e.g., the top-down representation). The probability(ies) can include a probability P, e.g., represented by a star encircling a portion (e.g., a pixel) of the top-down representation. The probability Pmay be a probability that an element of the output represents a drivable area. The probability Pmay be a probability that an element of the output represents a non-drivable area.

The machine learned (ML) modelcan output one or more heat maps. For example, a first heat map can include a probability that each individual pixel is a drivable area (e.g., drivable area). In such an example or another example, a second heat map can include a probability that each individual pixel is a non-drivable area (e.g., the non-drivable area).

In some examples, the output(s) from the machine learned (ML) modelcan be utilized to identify a boundary line. The boundary linecan include an outline of the drivable areadetermined based on the machine learned (ML) modelto control the vehicleas, and/or before, the vehicleenters the construction zone. The vehiclecan be controlled to traverse the drivable areabut not the non-drivable area.

While the outputs can include the output with the probability(ies) associated with the drivable area, as discussed above in the current disclosure, it is not limited as such. In some examples, the probability Pcan be utilized to determine at least one of an expanded drivable area (e.g., a drivable area with additional areas in comparison to the initial drivable area), or a non-incident area, in a similar way as for the drivable area. In some examples, the probability Pcan be utilized to determine at least one of an expanded non-drivable area (e.g., a non-drivable area with additional areas in comparison to an initial non-drivable area), or an incident area, in a similar way as for the non-drivable area. For instance, the incident area may represent one or more incident areas, including one or more blockades, one or more police blocking cards, one or more flare identified areas, one or more police state areas, one or more motorcades, one or more protestors, and so on, or any combination thereof.

At operation, the processcan include controlling a vehicle based on a trajectory. The trajectory, which can represent one or more trajectories associated with one or more vehicles, can be determined based on the output of the machine learned (ML) model. In some examples, the trajectory(ies) can include a trajectory utilized to control the vehicle (e.g., the vehicle). In those or other examples, the trajectory(ies) can include a trajectory utilized to control the vehicle.

The vehiclecan be controlled to navigate around objects associated with the zone(s) and/or the area(s). In some examples, the vehicle approaching the construction zone can be controlled to navigate through the construction zone, if appropriate, and/or around the construction zone. In those or other examples, the vehicleapproaching the construction zone can be controlled to navigate around, in front of, in back or, and/or adjacent to objects that maybe objects, such as cones, dividers, temporary curbs, signs, meters, flaggers, etc., or any combination thereof.

In some examples, controlling of the vehiclecan be triggered based on the construction zone information, such as information identifying the construction zone, and/or portions (e.g., cones, dividers, signs, etc.) of the construction zone on a roadway. In alternative or additional examples, controlling of the vehiclecan be triggered based on the parking destination map area information, such as information identifying the parking destination map area and/or portions (e.g., temporary curbs, signs, meters, etc.) of the parking destination map area on the roadway. For instance, the controlling of the vehiclecan be triggered in response to the identifying of the construction zone information and/or the parking destination map area information, such as by identifying the construction zone and/or the parking destination map area exists in front of the vehicle.

The vehiclecan be controlled to follow a dynamic object (e.g., the dynamic object) based on the moving velocity of the dynamic objectindicating that the dynamic objectis following a safe path. Characteristics, such as the velocity, associated with the dynamic objectmay be utilized by the machine learned (ML) modelto identify a path of the dynamic objectas a safe path. Because the dynamic objectis likely to proceed at a velocity above a threshold velocity only if the path of the dynamic objectis safe, the machine learned (ML) modelcan identify an area through which the dynamic objectis travelling as the drivable area.

An exampleillustrates the environment, with one or more trajectories being determined based on the probability(ies) output by the machine learned (ML) model. The vehiclecan determine, as part of the dynamic information, the trajectory(ies). In some examples, the trajectory(ies) can include a trajectoryutilized to control the vehicle. The vehiclecan be controlled based on the dynamic information.

The trajectorycan be determined based on the dynamic object. The trajectorycan be utilized to control the vehicleto follow the dynamic object. In those or other examples, the trajectory(ies) can include a trajectoryutilized to control the vehicle. The trajectorycan be utilized to control the vehicle. The vehiclecan be controlled to follow the dynamic object, based on the velocity of the dynamic objectbeing above the threshold velocity.

In some examples, the vehiclecan be controlled to follow the dynamic object, based on the velocity of the dynamic objectbeing above the threshold velocity and, possibly, based on the velocity of the dynamic objectnot being above the threshold velocity. In those or other examples, the vehiclecan be controlled to follow the dynamic object, based on the velocity of the dynamic objectbeing greater than the velocity of one or more other dynamic objects in, for instance, the non-drivable area. In those or other examples, the vehiclecan be controlled to follow the dynamic object, based on a difference between the velocity of the dynamic objectand individual velocities of one or more corresponding dynamic objects in the non-drivable areabeing greater than a threshold difference. In those or other examples, the vehiclecan be controlled to follow the dynamic object, based on a difference between the velocity of the dynamic objectand an average velocity of the individual velocities of the corresponding dynamic object(s) in the non-drivable areabeing greater than a threshold difference.

With respect to, a processis depicted in a flowchart for dynamic drivable area determining management. At an operation, the processcan include determining a dynamic object. A vehicle (e.g., the vehicle, as discussed above in further detail, or a similar vehicle) can receive sensor data and map data and process the sensor data and the map data to determine that a dynamic object is present in the environment, in a similar way as for determining the dynamic object at operation, as discussed above. In some examples, the operationcan include determining dynamic drivable area determining information (or “dynamic information”) associated with the environment. The dynamic information, for example, may be determined based on the dynamic object. In some examples, a dynamic object may represent one or more dynamic objects in one or more areas of the environment.

An exampleillustrates an environment including one or more vehicles, such as the vehicle. The environment illustrated in examplemay include one or more dynamic objects, such as dynamic objectsand. The environment that includes the vehicle, the dynamic object, and the dynamic objectmay be the same as, or different from the environment that includes the vehicleand the dynamic object.

An operation, the processcan include determining a top-down representation. The top-down representation can include any portion of the environment. For example, the top-down representation can be determined in a similar way as the top-down representation determined according to the operation, as discussed above.

An exampleillustrates the environment, with the top-down representation(s) that include various portions of the environment. The top-down representation(s) may include the vehicle, the dynamic object, the dynamic object, the zone(s), and/or the map area(s). For example, the top-down representation(s) can include a top-down representationwith the vehicle, the dynamic object, the dynamic object. The top-down representationmay include a parking destination map area.

Various characteristics associated with the dynamic object(s)and/orcan be utilized to identify the safe path. For example, one or more characteristics, such as one or more locations, one or more velocities, one or more directionalities of the dynamic object(s)and/orat one or more times, respectively, in the environment, can be utilized to control the vehicle. The characteristic(s) may inform a machine learned (ML) model (e.g., the machine learned (ML) model, as discussed below) that areas that otherwise would be non-parking locations are actually parking locations.

The top-down representationmay include different channels, such as different snapshots (e.g., dimensions) associated with different times (e.g., points in time) (e.g., temporal dimensions). The channels may be included in a multi-channel image of multi-channel image data with rasterized input (e.g., the rasterized input, as discussed above and below with reference to). For example, the temporal dimensions may be represented by a group of rectangles identified by reference numeral, as illustrated in.

The dynamic objectmay travel in front of, next to, in back of, etc., parking destination map area. The dynamic objectmay move around the parking destination map area. The dynamic objectmay have a velocity that is above a threshold velocity. The dynamic objectmay enter, remain in, and/or exit the parking destination area map. The dynamic objectmay have a velocity that is not above a threshold velocity.

At operation, the processcan include inputting the top-down representation (or “representation”) into a machine learned (ML) model. Inputting the representation can include inputting, into the ML model, the top-down representation, which can include the vehicle, the dynamic object, the dynamic object, and the parking destination map area.

An exampleillustrates the environment, with the top-down representation(s) being input into the machine learned (ML) model, such as a machine learned (ML) model. The machine learned (ML) modelmay represent one or more machine learned (ML) models, such as a machine learned model (ML). The top-down representation(s) being input into the machine learned (ML) modelcan include the top-down representationwith the vehicle, the dynamic object, the dynamic object, and the parking destination map area.

At operation, the processcan include receiving outputs with probabilities. The outputs can include an output with a probability that an element of the output represents a parking location. The output can include a first output; and the probability can include a first probability that an element of the first output represents the parking location. The outputs can include an output with a probability that an element of the output represents a non-parking location. The output can include a second output; and the probability can include a second probability that an element of the second output represents the non-parking location.

An exampleillustrates the environment, with one or more probabilities output by the machine learned (ML) model. The vehiclecan determine one or more outputs with the probability(ies) that one or more elements of the output(s) represent one or more parking locations and/or one or more non-parking locations. The output(s) may be included in the dynamic information.

The probability(ies) can include a probability P, e.g., represented by a star encircling a portion (e.g., a pixel) of the top-down representation (e.g., the top-down representation). The probability(ies) can include a probability P, e.g., represented by a star encircling a portion (e.g., a pixel) of the top-down representation. The probability Pmay be a probability that an element of the output represents a parking location. The parking location, for example, may include, and/or be represented as, polylines. Alternatively or additionally, the parking location, for example, may include, and/or be represented as rasterized image data. The probability Pmay be a probability that an element of the output represents a non-parking location.

The machine learned (ML) modelcan output one or more heat maps. For example, a first heat map can include a probability that each individual pixel is a non-drivable area (e.g., the parking location). In such an example or another example, a second heat map can include a probability that each individual pixel is a drivable area (e.g., the non-parking location).