System and method for detecting a trailer coupler and planning a trajectory

The present disclosure relates to advanced driver assistance systems (ADAS), and more particularly to driver assistance systems that detect a trailer coupler used to couple a trailer to a vehicle and that provide trajectory planning and navigation assistance.

Vehicles are often used to tow trailers to carry equipment, goods, belongings, animals, other vehicles including boats and off-road vehicles, from place to place along roadways. Vehicles are equipped with hitches that removably couple to a trailer coupler of a trailer. Coupling the trailer coupler to a vehicle hitch can be difficult and require backing up the vehicle with limited operator vision.

Accordingly, trailer coupler identification systems have been developed to assist in locating the trailer coupler on the trailer. For example, some ADAS employ an electronic tag, marker, or ID placed on the trailer coupler that is easily identifiable by the vehicle's computer vision system of the ADAS. ADAS that assist with backing are often only effective at close range (e.g., <3 feet), require a marker, and often only utilize one camera in the computer vision system. While current systems and methods for trailer coupler detection and navigation assistance achieve their intended purpose, there is a need for a system and method of trailer coupler detection and navigation assistance that improves the accuracy of locating the trailer coupler on a trailer when backing a vehicle, reducing computational burdens, and reducing component complexity while decreasing trailer coupling difficulty, and thereby improving operator confidence and satisfaction.

According to several aspects of the present disclosure, a system for detecting a trailer coupler on a trailer to assist with coupling the trailer coupler to a vehicle hitch is provided. The system includes at least one human-machine interface (HMI) disposed within the vehicle, a vehicle camera disposed on the vehicle and detecting an environment surrounding the vehicle, a trailer camera disposed on the trailer and detecting an environment surrounding the trailer, a vehicle-to-infrastructure (V2I) communication network, a mobile device application (App) with access to camera and mobile network and one or more controllers. Each of the one or more controllers has a processor, a memory, and one or more input/output (I/O) ports, the I/O ports in communication with the vehicle camera and the trailer camera, and the one or more HMIs via the V2I communication network. The memory stores instructions including a trailer detection and trajectory planning application (DTPA). The DTPA includes instructions that initialize a data stream from the vehicle camera and the trailer camera that can be a mobile device with camera running the OEM App positioned in the trailer, simultaneously capture the data stream from the vehicle camera and from the trailer camera, determine key landmarks in the data stream from the vehicle camera and in the data stream from the trailer camera, combine the key landmarks, determine a coupler location with respect to the vehicle hitch using combined key landmarks, compute a joint reference trajectory using the coupler location with respect to the hitch location, and provide navigation assistance in real time based on the joint reference trajectory.

In another aspect of the present disclosure, the trailer camera is a cell phone camera in communication with a mobile application.

In yet another aspect of the present disclosure, the one or more controllers is located in the vehicle and at the trailer.

In yet another aspect of the present disclosure, the one or more controllers is located at a remote computing system.

In yet another aspect of the present disclosure, the system includes instructions that further request an operator to provide an input to the HMI to confirm initialization of the DTPA.

In yet another aspect of the present disclosure, the system includes instructions that perform key landmark detection and dimensional calculations on the captured data stream to determine the location of the trailer coupler in real time.

In yet another aspect of the present disclosure, the system further includes instructions for presenting navigation assistance including at least one of verbal or visual navigation assistance to the HMI to guide the vehicle hitch to the trailer coupler.

In yet another aspect of the present disclosure, the system further includes instructions for providing navigation assistance including automated steering assistance.

In yet another aspect of the present disclosure, a method for detecting location of a trailer coupler on a trailer and assisting with the coupling of the trailer coupler to a vehicle hitch on a vehicle is provided. The method includes initializing a data stream from the vehicle camera and the trailer camera, capturing the data stream from the vehicle camera and from the trailer camera, determining key landmarks in the data stream from the vehicle camera and in the data stream from the trailer camera, combining the key landmarks, determining a coupler location with respect to the vehicle hitch using combined key landmarks, computing a joint reference trajectory using the coupler location with respect to the hitch location, and providing navigation assistance in real time based on the joint reference trajectory.

In yet another aspect of the present disclosure the method further includes requesting an operator to provide an input to the HMI.

In yet another aspect of the present disclosure the method further includes providing a remote computing system, wherein the I/O ports are in communication with the remote computing system via the V2I communication network.

In yet another aspect of the present disclosure the method further includes capturing the data stream while simultaneously capturing video stream from the vehicle camera and from the trailer camera.

In yet another aspect of the present disclosure the method includes presenting the navigation assistance in real time and presenting at least one of verbal or visual assistance using the one or more human-machine interfaces.

In yet another aspect of the present disclosure the method further includes presenting the navigation assistance in real time and providing steering assistance.

In yet another aspect of the present disclosure the method further includes determining key landmarks in the data stream from the vehicle camera and in the data stream from the trailer camera using the processor in the vehicle.

In yet another aspect of the present disclosure the method further includes determining key landmarks in the data stream from the vehicle camera and in the data stream from the trailer camera using a processor in the vehicle and in a processor in the trailer.

In yet another aspect of the present disclosure the method further includes determining key landmarks in the data stream from the vehicle camera and in the data stream from the trailer camera using the controller in a remote computer.

In yet another aspect of the present disclosure the method includes the trailer camera being a mobile device camera located at the trailer coupler.

In yet another aspect of the present disclosure, a method for detecting location of a trailer coupler on a trailer and assisting with the coupling of the trailer coupler to a vehicle hitch on a vehicle is provided. The method includes requesting an operator to provide an input to the HMI, initializing a data stream from the vehicle camera and the trailer camera, capturing the data stream from the vehicle camera and from the trailer camera, determining key landmarks in the data stream from the vehicle camera and in the data stream from the trailer camera in real time, combining the key landmarks determined from the data stream from the vehicle camera and from the data stream from the trailer camera, and determining a coupler location with respect to the vehicle hitch using combined key landmarks. The DTPA also includes control logic for: actively and dynamically computing a joint reference trajectory using the coupler location with respect to the hitch location, and actively and continuously presenting navigation assistance in real time based on the joint reference trajectory, wherein the navigation assistance includes automated steering assistance.

In yet another aspect of the present disclosure the method includes navigation assistance with verbal and visual instruction displayed on the one or more HMIs.

Further areas of applicability will become apparent from the description provided herein. It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.

The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

Referring to, a schematic diagram of a systemfor detecting a trailer coupler and planning a trajectory from a vehicle hitch to the trailer coupler is shown. The systemincludes a vehicleand a trailerattachable via a hitchto the vehicle. While the vehicleis shown as a passenger car or sport utility vehicle (SUV), it should be appreciated that the vehiclemay be any type of vehicle without departing from the scope or intent of the present disclosure. In some examples, the vehiclemay be a car, a truck, an SUV, a bus, a semi tractor, a tractor used in farming or construction or the like, a pushback tractor or tug, a baggage tractor or cargo tow tractor, or the like. Likewise, while the trailershown inis a single-axle trailer, the trailermay be any of a wide variety of trailertypes without departing from the scope or intent of the present disclosure. In some examples, the trailermay be an enclosed or open-air trailerincluding flatbed trailers, dry vans, refrigerated trailers, lowboy trailers, fifth wheel campers, gooseneck trailers, luggage or cargo trailers, or the like. In further examples, the trailermay be another vehicleas described above, or a vehiclesuch as a plane, helicopter, or other such aircraft. That is, a vehiclemay tow another vehicleas a trailer, and multiple trailersmay be towed together in single file, or in parallel with one another.

The traileris attachable or mountable to the vehiclevia a hitchon the vehicleand a trailer coupleron the trailer. The hitchand trailer couplerallow rotational movement of the trailerrelative to the vehicle, thus providing the trailerand vehiclemeans to negotiate turns while in motion. The hitchdepicted inis a ball hitch, however it should be appreciated that other forms of hitchesmay be used without departing from the scope or intent of the present disclosure. For example, the hitchmay be a ball hitch, a receiver hitch, a fifth wheel hitch, a gooseneck hitch, a pintle hitch, a bumper hitch, a weight distribution hitch, or the like. In several aspects, the hitchoperates as a pivot. It is desirable while attempting to couple the trailer to the vehicle to manage vehiclesuch that the vehicledoes not physically contact the trailer or obstacles in the environmentsurrounding the vehicleand trailerwhen the vehicleis in reverse and backing towards the trailer.

The systemincludes one or more controllers. The controlleris a non-generalized, electronic control device having a preprogrammed digital computer or processor, non-transitory computer readable medium or memoryused to store data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and one or more input/output (I/O) ports. Computer readable medium includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory. The non-transitory computer readable medium or memoryincludes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device. Computer code includes any type of program code, including source code, object code, and executable code. The processoris configured to execute the code or instructions. In some examples, the controllermay be a dedicated Wi-Fi controller or an engine control module, a transmission control module, a body control module, an infotainment control module, etc. The I/O portsare configured to communicate via wired or wireless connections using Wi-Fi protocols under IEEE 802.11x, Bluetooth communication protocols, radio frequency (RF) protocols, or the like. In some examples, the controllerfurther includes one or more applications. An applicationis a software program configured to perform a specific function or set of functions. The applicationmay include one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in a suitable computer readable program code. The applicationmay be stored within the memoryor in additional or separate memory. Examples of applicationsinclude audio or video streaming services, games, browsers, social media, suspension and engine control programs, body control programs, advanced driver assistance system (ADAS) programs, and the like. In a particular applicationof the present disclosure, the systemincludes a trailer detection and trajectory planning application (DTPA).

Systemfurther includes one or more sensors. The sensorsproduce real-time positioning and detection information regarding the location and/or movement of the vehicle. In the example provided, the sensorsinclude vehicle camerasand trailer camera′. The vehicle camerais fixed to a rear of the vehiclethat has a field of view to the rear of the vehicle. Alternatively, the vehicle cameramay be located on a side view or rear-view mirror of the vehicle. The trailer camera′ may be located at the coupleror other location of the trailer. However, the sensorsmay include any of a wide variety of different types of sensors, including but not limited to: Light Detection and Ranging (LiDAR) sensors, Radio Detection and Ranging (RADAR) sensors, Sound Navigation and Ranging (SONAR) sensors, ultrasonic sensors, or combinations thereof. Further, the sensorsmay have the ability to communicate with a Global Positioning System (GPS), and in particular, image data collected by satellitesin orbit around Earth, in order to more accurately and precisely report the location of the vehicle. In further examples, the sensorsmay include wheel speed sensors disposed on one or more of the vehicleand the trailer, or a hitch angle estimation sensor usable with a hitch angle estimation algorithm.

In additional examples, the sensorsmay include inertial measurement units (IMUs). IMUs measure and report attitude or position, linear velocity, acceleration, and angular rates relative to a global reference frame using a combination of some or all the following: accelerometers, gyroscopes, and magnetometers. In some examples, IMUs may also utilize global positioning system (GPS) data to indirectly measure attitude or position, velocity, acceleration, and angular rates. When used in the systemof the present disclosure, the IMUs measure and report attitude or position, linear velocity, acceleration, and angular rates the vehicleand vehicle camera. The controllersand sensorsmay be linked to and communicate on one or more communications networksincluding one or more of a vehicle-to-vehicle (V2V) communications network, a vehicle-to-infrastructure (V2I) communications network(a remote computing system), or the like. For the sake of brevity, the term “infrastructure” is used to collectively refer to network entities capable of Ad hoc wireless communications in the direct link or side link, and the cellular communication network owned by service providers, hence vehicle-to-network (V2N) communication is also intended to be included in the scope of V2I.

Referring now toand with continuing reference to, the systemfurther includes a human-machine interface (HMI)disposed within the vehicleto interact with an operator of the vehicle. In several aspects, the HMIincludes one or more devices capable of interacting with the operator, such as a screen disposed within the vehiclesuch as an instrument cluster, an infotainment screen, a heads-up display (HUD), an interior rear-view screen such as a rear-view mirror augmented by a screen, a sound delivery system, speakers, a microphone, or the like. However, it should be appreciated that other HMIsare considered herein as well. For example, the HMImay be a mobile device, such as a tablet computer, a mobile phone, a cell phone in communication with the DTPA and running an original equipment manufacturer (OEM) app positioned at the trailer, or the like, and the HMImay be provided by the operator and temporarily mounted to or disposed on an interior passenger compartment component of the vehicle. In several aspects, the HMIcommunicates with the controllersvia the I/O portsand sends information to and receives information from the I/O portsof the controllers, including during operation of the DTPA.

For example, the DTPAin combination with other vehicle control applications may assist a vehicleoperator in maneuvering the vehicleby providing verbal (e.g., via a vehicle speaker) or visual instruction (e.g., via HMI, HUD, and the like) to the operator or causing a steering wheelto rotate (e.g., automated steering assistance) and direct the vehicletowards traileras it moves toward the trailerto attach, hitch or couple the trailerto the vehicle. The verbal and/or visual assistance can be configured by an operator in the HMIor an application (e.g., OnStar) in-vehicleand/or an application at the trailer(e.g., a mobile phone using OnStar). More specifically, the DTPAincludes a plurality of sub-routines or instructions that are stored in memoryof the controllersand executed by the processorwhile data is received, via the I/O ports, from the sensorssuch as the vehicle cameraand the trailer camera′, hitch angle estimation sensor, IMUs, and/or satellitesreporting GPS data. The DTPAfurther includes a plurality of subroutines or instructions that cause data to be transmitted from the controllersto the HMI.

With reference now to, a flowchart of a methodof the DTPAis shown, in accordance with the present disclosure. The DTPAis initialized or started at blockwhen one or more specific conditions occur. The specific conditions include manual initialization by the operator via the HMIor the operation of a physical button disposed in the interior of the vehicle. In further examples, the conditions may be automatically satisfied by sensorsand/or the vehicle cameraand the trailer camera′ detecting that a traileris proximate but not coupled to the vehicle. The methodthen proceeds to block.

At block, upon receiving confirmation at blockthat the operator desires to engage the DTPA, the controllerutilizes the HMIto present a prompt to the operator for confirmation that the operator desires that the DTPAbe engaged or to request that the operator provide an input to the HMI. The prompt may be displayed on-screen on the HMI, HUD, dictated audibly through a stereo system of the vehicle, or the like. The methodthen proceeds to block.

At block, the data stream starts and is received by the controllerfrom the vehicle cameraand the trailer camera′. The data stream includes video data or image frames from the video data, graphic data, or other data from the vehicle cameraand the trailer camera′. The controller receives the data stream simultaneously and in real time from both the vehicle cameraand the trailer camera′. The method then proceeds to block.

At block, the DTPAuses a visual simultaneous localization and mapping (vSLAM, Co-SLAM, Multi-Co-SLAM) algorithm that uses visual, video, and/or optical data to capture the data stream from the vehicle cameraand the trailer camera′ and determine key landmarks. Environmental and key landmark scan data received from the vehicle cameraand the trailer camera′ is processed in one or more image or video-processing algorithms (e.g., vSLAM). In several aspects, the image or video-processing algorithms may include one or more of image capture algorithms that capture an image of the trailerand trailer couplerfrom the video data stream and perform image distortion corrections as needed to correct for camera,′ lens shape, and the like. Feature or landmark detection algorithms, such as Canny Edge algorithms, performs feature or landmark detection within the image of the trailerand trailer coupler.

With continuing reference toand now with additional reference towhich shows exemplary camera poses as the vehiclemoves over time, the DTPAuses, for example, a vSLAM algorithm to determine and reconstruct structure and key landmarks in environmentusing a data stream from the vehicle cameraand from the trailer camera′. While vSLAM is used throughout the disclosure, it should be understood that other image or video-processing algorithms and/or visual simultaneous localization and mapping applications may be utilized. The vSLAM algorithm uses the optical and/or video data received at blockto perform camera pose calculations. Key landmarks include landmarks (e.g., hitch, coupler, and the like) that are relevant to determining location of the vehiclewith respect to the trailer. Environmentincludes structure, key landmarks, and surroundings from a viewpoint of the vehicle cameraand trailer camera′.

Referring to, for example, a first camera pose, a second camera pose, and a third camera poseare shown as the vehicleand the camera,′ move over time T1, T2, T3. The camera poses,,are captured from the data stream from vehicle cameraand the trailer camera′. The vSLAM algorithm provides estimates for the position and orientation of the cameraon vehiclewith respect to the environmentand with respect to the camera′ on trailerwhile simultaneously mapping the environment. Vehicle cameraand the trailer camera′ capture images of the environmentso that the vSLAM algorithm can extract features and key landmarks, such as the trailer couplerand hitch. The vSLAM algorithm also performs loop closure detection for recognizing that the vehicle cameraand the trailer camera′ are returning to a previously captured environmentand corrects the drift in the map and the camera pose.

Returning to, once the key landmarks are determined at block, the method proceeds to blockwhere the key landmarks from the data stream captured in blockare combined. The key landmarks determined at block, for example the hitchand the trailer coupler, are combined and mapped across consecutive video frames to estimate the motion of the camera,′ and the 3D position of the hitchand the trailer couplerand/or other features in the environment. The camera poses,,captured by camera,′ and the 3D features are combined to form a map of the environmentthat is updated as new images are captured and processed by systemin real time (e.g., actively and continuously).

For example, with continuing reference toand now with reference to, optical or video data (e.g., an image or camera pose from a video frame) from vehicle camerais received and is shown having identified landmarks u, i. Additionally, optical or video data (e.g., an image or camera pose from a video frame) from trailer camera′ is received and is shown having identified landmarks v, k, j. Landmark u, determined from an image from the vehicle camera, is mapped to landmark v, which is determined from an image from the trailer camera′. Landmark i is not mapped to landmark k because the vSLAM algorithm determines that mapping landmark k to landmark i is not feasible or likely. A landmark that is not feasible is determined, for example, when a first landmark image (e.g., landmark i) does not match a second landmark image (e.g., landmark k). Instead, landmark j is mapped to landmark i, and key landmarks u and v are mapped and combined and key landmarks land j are mapped and combined.

Returning to, at blockthe controllerdetermines a location of the couplerwith respect to the vehicle hitchusing the combined key landmarks from block. The location of the coupleris estimated using the combined key landmarks in an X, Y, Z coordinate system by measuring the vehicle dynamics as the vehiclemoves over time (α=pitch, β=roll, γ=yaw and angular rotations thereof). For example, the vSLAM algorithm is used to locate the trailer couplerin each of the camera poses,,shown infor each time T1, T2, and T3, thus projecting coupler 3D position in each of the images of each of the camera poses,,to a 2D image. Shown in, location of the hitchrelative to camera′ and the location of the couplerwith respect to vehicle cameraare determined using a calculated distance between camera,′ and “p1” (e.g., a determined landmark) given that both distance “a” and distance “b” are known. Distance “a” is the vertical height of the camerain the vehicle and distance “b” is the horizontal distance from the camera,′ to p1. The methodthen proceeds to block.

At block, the DTPAcomputes an optimal joint reference trajectory RTusing the location of the couplerdetermined at block. A reference trajectory RTfrom a perspective of the vehicle camerais computed, and a reference trajectory RTfrom a perspective of the couplerand trailer camera′ is computed. A reference trajectory RTis then computed (using, for example, Euclidean distance, a dynamic time warping algorithm, and the like) by using absolute value of the coupler reference trajectory RTsubtracted from the vehicle reference trajectory RT(i.e., (RT=|RT−RT|) using DTPA. An optimal reference trajectory RTfrom the hitchto the coupleris then computed so that RTis minimal. Computing of the optimal joint reference trajectory RTcan be repeated until the vehicleis coupled with the trailer. The optimal joint reference trajectory RTis computed using the DTPAwhere the controlleris located in the vehicle, the trailer, and/or network,(e.g., a remote computer). The methodthen proceeds to block.

At block, the DTPAprovides navigation assistance in real time based on the joint reference trajectory RTcomputed at block. The DTPAprovides navigation assistance in the form of verbal or visual instructions to an operator via the HMIor HUDfor directing and steering the vehicle. For example, visual instructions can be displayed on the HMIindicating a direction to turn a steering wheel and indicating a speed for the operator to operate the vehicle. In another aspect, the DTPAprovides navigation assistance by providing navigation instructions to and directing an automated or a semi-automated steering system having an actuator to rotate a passenger vehicle steering wheel. The methodthen ends.

The systemand methodfor detecting the trailer couplerproviding trajectory planning and navigation assistance of the present disclosure offers several advantages over prior art systems. These include: detecting a couplerfrom a range of greater than 3 feet, eliminating the need for a couplerwith a marker by using a trailer camera′, and using bi-directional detection and trajectory planning, thereby improving vehicle and trailer coupling for an operator.

The description of the present disclosure is merely exemplary in nature and variations that do not depart from the gist of the present disclosure are intended to be within the scope of the present disclosure. Such variations are not to be regarded as a departure from the spirit and scope of the present disclosure.