Method for verifying accuracy of virtual sensor model for simulation based on reality information data

This application is based on and claims priority under 35 U.S.C. § 119 to Korean Patent Application No. 10-2022-0173316, filed on Dec. 13, 2022, in the Korean Intellectual Property Office, the disclosure of which is herein incorporated by reference in its entirety.

The disclosure relates to an autonomous driving simulation, and more particularly, to a method for verifying accuracy of a virtual sensor model which is used for verifying an autonomous driving algorithm in a virtual road environment by using a simulator.

An autonomous vehicle may have a sensor mounted therein to perform recognition, determination, and control, and may sense surroundings of the ego vehicle and may identify information of omnidirectional objects. In particular, an autonomous vehicle may recognize its surroundings, such as objects around the vehicle, a drivable region on a road, a traffic signal state, etc., from sensor data of a camera, a LiDAR, a RADAR, etc. mounted therein, may determine a moving direction of a neighboring object, a current drivable status, etc., and may control a movement of the ego vehicle.

In order to verify an autonomous driving algorithm mounted in an autonomous vehicle, a simulator-based method may be used in parallel with real vehicle-based verification for rapid and effective verification. Various sensor data required to perform operations of a recognition, determination, and control algorithm mounted in an autonomous vehicle may be generated by modeling and simulating a virtual sensor, and may be provided to the algorithm. Accordingly, it is very important to maintain accuracy of an output value of a virtual sensor model which is used in an autonomous driving simulator for accurate operations of an algorithm.

The disclosure has been developed in order to solve the above-described problems, and an object of the disclosure is to provide a method for measuring and verifying accuracy of virtual sensor data which is supplied to a recognition, determination, control algorithm for autonomous driving when autonomous driving software mounted in an autonomous vehicle is verified based on a simulator.

According to an embodiment of the disclosure to achieve the above-described object, a virtual sensor verification method may include: acquiring information on positions and states of real vehicles which are running on a real road; acquiring real sensor data generated in real sensors of a reality information acquisition vehicle from among the real vehicles; reproducing the real vehicles on a virtual road as virtual vehicles, based on the acquired information on the positions and states; acquiring virtual sensor data outputted from virtual sensors mounted in a virtual information acquisition vehicle from among the virtual vehicles; and verifying the virtual sensors by comparing the acquired real sensor data and the virtual sensor data.

The virtual sensors may be virtual sensors that simulate types and specifications of the real sensors.

The real sensors may include a real camera, a real LiDAR, and a real RADAR, and the virtual sensors may include a virtual camera, a virtual LiDAR, and a virtual RADAR.

The information on the positions and states of the real vehicles may be acquired from GNSS/INS mounted in the real vehicles.

The GNSS/INS mounted in the real vehicles may be synchronized with reference to a GPS time of GNSS/INS mounted in the reality information acquisition vehicle.

A difference between a GPS time of the reality information acquisition vehicle and a GPS time of a target vehicle may be calculated by the following equation:

where tis a GPS time difference,

is a GPS time data processing delay time at acquisition equipment of the reality information acquisition vehicle,

is a GPS time data processing delay time at acquisition equipment of the target vehicle,

is a GPS reception delay time at acquisition equipment of the reality information acquisition vehicle, and

is a GPS reception delay time at acquisition equipment of the target vehicle.

The real sensor data of the reality information acquisition vehicle may be synchronized with reference to a GPS time of the reality information acquisition vehicle.

The virtual road may be a road that simulates a real road in a virtual space.

A simulator may be a tool for testing an autonomous driving algorithm through the reality information acquisition vehicle.

According to another aspect of the disclosure, a virtual sensor verification system may include: a synchronization module configured to acquire information on positions and states of real vehicles which are running on a real road, and to acquire real sensor data generated in real sensors of a reality information acquisition vehicle from among the real vehicles; a simulation module configured to reproduce the real vehicles on a virtual road as virtual vehicles, based on the acquired information on the positions and states; and a verification module configured to acquire virtual sensor data outputted from virtual sensors mounted in a virtual information acquisition vehicle from among the virtual vehicles, and to verify the virtual sensors by comparing the acquired real sensor data and the virtual sensor data.

According to still another aspect of the disclosure, a virtual sensor verification method may include: reproducing real vehicles on a virtual road as virtual vehicles, based on information on positions and states which is acquired from real vehicles running on a real road; acquiring virtual sensor data outputted from virtual sensors mounted in a virtual information acquisition vehicle from among the virtual vehicles; and verifying the virtual sensors by comparing the acquired virtual sensor data and real sensor data which is acquired by real sensors of a reality information acquisition vehicle among the real vehicles.

According to yet another aspect of the disclosure, a virtual sensor verification system may include: a simulation module configured to reproduce real vehicles on a virtual road as virtual vehicles, based on information on positions and states which is acquired from real vehicles running on a real road; and a verification module configured to acquire virtual sensor data outputted from virtual sensors mounted in a virtual information acquisition vehicle from among the virtual vehicles, and to verify the virtual sensors by comparing the acquired virtual sensor data and real sensor data which is acquired by real sensors of a reality information acquisition vehicle among the real vehicles.

According to embodiments of the disclosure as described above, accuracy of virtual sensor data which is supplied to a recognition, determination, control algorithm for autonomous driving may be measured and verified, so that accuracy on a result of verifying based on a simulator of an autonomous driving algorithm may be enhanced.

According to embodiments of the disclosure, it is possible to verify autonomous driving software based on a scenario of various environments and conditions, so that algorithms may be rapidly developed and stability of algorithms may be enhanced.

Other aspects, advantages, and salient features of the invention will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses exemplary embodiments of the invention.

Before undertaking the DETAILED DESCRIPTION OF THE INVENTION below, it may be advantageous to set forth definitions of certain words and phrases used throughout this patent document: the terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation; the term “or,” is inclusive, meaning and/or; the phrases “associated with” and “associated therewith,” as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like. Definitions for certain words and phrases are provided throughout this patent document, those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

Hereinafter, the disclosure will be described in more detail with reference to the accompanying drawings.

An embodiment of the disclosure proposes a method for verifying accuracy of a virtual sensor model for simulation based on reality information data. The disclosure relates to a technology for measuring and verifying accuracy of a virtual sensor model which is used for verifying an autonomous driving algorithm in a virtual road environment using a simulator.

is a view illustrating a configuration of a virtual sensor model verification system according to an embodiment of the disclosure. The virtual sensor model verification system according to an embodiment may include a sensor data synchronization module, an autonomous driving simulation module, a virtual sensor verification moduleas shown in.

The sensor data synchronization modulemay receive data from real vehicles,-,-, . . . ,-that are running on a real road, and may synchronize data. The real vehicles,-,-, . . . ,-may be divided into a reality information acquisition vehicleand target vehicles-,-, . . . ,-

The reality information acquisition vehicleis an ego vehicle which is an object for generating real sensor data, and the target vehicles-,-, . . . ,-refer to vehicles thar are running around the reality information acquisition vehicle.

Data collected from the reality information acquisition vehiclemay include global navigation satellite system (GNSS)/inertial navigation system (INS) data and real sensor data. On the other hand, the target vehicles-,-, . . . ,-provide only GNSS/INS data.

The GNSS/INS data may contain information on positions and states (positions, direction, speeds of vehicles, etc.) of the real vehicles,-,-, . . . ,-. Real sensors installed in the reality information acquisition vehiclemay include a camera, a LiDAR, a RADAR, and may further include other types of sensors.

Accordingly, the sensor data synchronization modulemay receive GNSS/INS data and real sensor data from the reality information acquisition vehicle, and may receive GNSS/INS data from the target vehicles-,-, . . . ,-, and may synchronize the received data. A method for synchronizing by the sensor data synchronization modulewill be described in detail below with reference to.

The autonomous driving simulation modulemay reproduce the real vehicles,-,-, . . . ,-on a virtual road as virtual vehicles, based on information on positions and states of the real vehicles,-,-, . . . ,-which is recorded on the GNSS/INS data transmitted through the sensor data synchronization module.

An autonomous driving simulation is a tool for testing an autonomous driving algorithm in a virtual environment through a virtual vehicle. The virtual road is a road in a virtual space, which simulates a real road.

A virtual information acquisition vehicle which is a virtual vehicle corresponding to the reality information acquisition vehicleamong the virtual vehicles may have virtual sensor models mounted therein. The virtual sensor models are virtual sensors that simulate real sensors installed in the reality information acquisition vehicleto be of the same type and to have the same specification.

The virtual sensor verification modulemay verify accuracy of a virtual sensor model mounted in the virtual information acquisition vehicle by comparing real sensor data and virtual sensor data. Real sensor data may be acquired from the reality information acquisition vehiclethrough the sensor data synchronization module, and virtual sensor data may be acquired from the autonomous driving simulation module.

is a flowchart provided to explain a virtual sensor model verification method according to another embodiment.

In order to verify a virtual sensor model, the sensor data synchronization modulemay acquire information on positions and states of the real vehicles,-,-, . . . ,-running on a real road (S), first, and may acquire real sensor data from the reality information acquisition vehicle(S).

The autonomous driving simulation modulemay reproduce the real vehicles,-,-, . . . ,-on a virtual road as virtual vehicles, based on the information on the positions and states acquired at step S(S).

The virtual sensor verification modulemay acquire virtual sensor data from a virtual sensor model mounted in a virtual information acquisition vehicle, which corresponds to the reality information acquisition vehicle(S), and may verify accuracy of the virtual sensor model by comparing the acquired virtual sensor data with the real sensor data which is acquired at step S(S).

Hereinafter, a method for synchronizing by the sensor data synchronization moduledescribed above will be described in detail.

Data synchronization may be required to implement a virtual environment based on data acquired from the reality information acquisition vehicleand the target vehicles-,-, . . . ,-. A method for synchronizing data acquired from different real vehicles is illustrated in.

It is assumed that GNSS/INS data of a neighboring target vehicle are acquired with reference to a GPS time of the reality information acquisition vehicleas shown in. There is a high possibility that a time at which GNSS/INS data is acquired from the reality information acquisition vehicleand a time at which GNSS/INS data is acquired from the neighboring target vehicle differ from each other. In addition, there may be a difference in time of starting to acquire GNSS/INS data, and accordingly, a process of synchronizing data with reference to the reality information acquisition vehicle, which is an ego vehicle, is required.

illustrates an example of GNSS/INS data which is acquired by data acquisition equipment mounted in the reality information acquisition vehicleand data acquisition equipment mounted in the neighboring target vehicle. In order to compare by UTC time, time information of each piece of data acquisition equipment may be acquired, and GPS time data may be acquired.

A method of calculating tto extract accurate comparison data by synchronizing sensor data acquired from different pieces of equipment is based on the following equation:

In the above equation, data