Video data generation method and device superimposing driver's frame of view and text or icon indicating driver's motion on vehicle exterior video

This application is based on and claims priority under 35 USC 119 from Japanese Patent Application No. 2023-57993 filed on Mar. 31, 2023, the entire content of which is incorporated herein by reference.

Embodiments disclosed herein relate to a generation method, a display device, and a generation device.

There has been known a technique in the related art in which, when a vehicle exterior video acquired by a drive recorder or the like is reproduced, an angle of view of a driver is obtained based on vehicle speed information acquired together with the vehicle exterior video, and a range of view corresponding to the angle of view is displayed together with the vehicle exterior video (for example, refer to JP2009-089022A).

Such a technique in the related art is used to, for example, verify a cause of an accident and a near-miss.

In the above-described technique, the angle of view is obtained and merely the range of view of the driver is displayed together with the vehicle exterior video.

For this reason, the above-described technique does not allow a viewer of the vehicle exterior video to grasp an actual field of view of the driver or what led the driver to the field of view.

Aspects of the present disclosure relates to providing a generation method, a display device, and a generation device that allow a viewer of a vehicle exterior video to grasp an actual field of view of a driver and what led the driver to the field of view.

According to an aspect of the present disclosure, there is provided a generation method by a controller, the generation method including: acquiring a vehicle interior video from an in-vehicle camera and a vehicle exterior video from an out-vehicle camera; executing a motion analysis of feature points of at least one of a face, an eye, or a hand of a driver based on the vehicle interior video; and generating video data displaying a frame of view of the driver and motion information of the driver superimposed on the vehicle exterior video based on the action analysis.

According to an aspect of the present disclosure, it is possible to allow a viewer of a vehicle exterior video to grasp an actual field of view of a driver and what led the driver to the field of view.

Hereinafter, an embodiment of a generation method, a display device, and a generation device disclosed in the present application will be described in detail with reference to the accompanying drawings. The present invention is not limited to the following embodiment.

In the following description, a case in which a generation system according to the embodiment is a teaching material video generation system(seeor the likes) will be described as an example. The teaching material video generation systemgenerates a teaching material video, which is a teaching material for e-learning for safe driving guidance or the like.

In the following description, the generation device according to the embodiment is a server device(seeand the subsequent drawings) provided in the teaching material video generation system. The generation method according to the embodiment is a teaching material video generation method executed by the server device.

In the following description, expressions “specific”, “prescribed”, and “constant” may be read as “predetermined”.

First, an outline of the teaching material video generation method according to the embodiment will be described with reference to.is a first schematic diagram of the teaching material video generation method according to the embodiment.is a second schematic diagram of the teaching material video generation method according to the embodiment.

As shown in, the teaching material video generation systemincludes a drive recorder, the server device, and a participating device(-,-, . . . ).

The drive recorderis a video recording device mounted on a vehicle. The drive recorderaccording to the embodiment includes an in-cameraand an out-camera. The in-cameraand the out-cameraare examples of a “camera”. The in-camerais configured to capture a vehicle interior video. The out-camerais configured to capture a vehicle exterior video.

When the vehicle is running, the drive recorderrecords vehicle data, which includes video data captured by the in-cameraand the out-camera, in a circular buffer memory in an over-writable manner for a fixed period (for example, 24 hours). The vehicle data may include, in addition to the video data, various types of data indicating a situation of the vehicle such as time information, position information, vehicle speed, and a G-value.

The drive recorderis configured to detect a specific event such as an accident and a near-miss. The drive recorderdetects the specific event when a change in the vehicle speed, a change in the G-value and the like satisfy prescribed event conditions corresponding to an accident, a near-miss and the like that are set in advance.

When detecting the specific event, the drive recordersets vehicle data for a prescribed period before and after a detection time point to be overwrite-prohibited. Alternatively, the drive recorderrecords the vehicle data for the prescribed period before and after the detection time point in another recording medium. The overwrite-prohibition process and the recording process on another recording medium may be executed in response to instructions from the server device.

When detecting the specific event, the drive recordertransmits the vehicle data set to be overwrite-prohibited to the server device.

The server deviceis provided to analyze the situation of the vehicle when the specific event such as an accident and a near-miss is detected based on the vehicle data transmitted from the vehicle. The server deviceis configured to generate a teaching material video based on an analysis result.

The server deviceis further configured to provide a service of delivering the generated teaching material video to the participating device. The participating deviceis a terminal device used by a participant (corresponding to an example of a “viewer”) participating in e-learning. The participating deviceis implemented by a personal computer (PC) such as the participating device-or a smartphone such as the participating device-.

There has been a technique in which an angle of view of a driver is obtained based on vehicle speed acquired together with video data from the drive recorderand a range of view corresponding to the angle of view is displayed together with a vehicle exterior video. Here, a case of using the technique in the related art to generate a teaching material video is considered.

However, in the above-mentioned technique, the range of view, which is defined by a correspondence map indicating a correspondence relation between the speed and the angle of view defined in advance, is merely displayed together with the vehicle exterior video.

The teaching material video generated using the above-mentioned technique cannot have the participant grasp an actual field of view of the driver and what led the driver to the field of view. Specifically, the participant cannot grasp whether a cause of an accident, a near-miss and the like is dozing, operating smartphone (hereinafter referred to as “phone”), looking aside or the like.

This is not limited to the case in which the participant views the teaching material video, and the same applies to a case in which a person in charge of analyzing a situation of a vehicle during an event (corresponding to an example of the “viewer”) views a verification video generated using the technology in the related art.

In the teaching material video generation method according to the embodiment, the server device(corresponding to an example of a “generation device”) acquires the vehicle interior video and the vehicle exterior video from the in-cameraand the out-camera

The server devicefurther executes an action analysis of feature points of a face, eyes, or a hand of the driver based on the vehicle interior video, and generates video data displaying a frame of view of the driver and an action of the driver together with the vehicle exterior video based on a result of the action analysis.

Specifically, as shown in, when detecting a specific event based on an event condition set in advance (step S), the drive recorder first transmits vehicle data for a constant period before and after an event detection time point to the server device(step S). As described above, the vehicle data includes video data, time information (occurrence date and time), position information, vehicle speed, G-value, and the like.

The server devicethen executes an action analysis of the feature points of the face, the eyes, or the hand of the driver based on the vehicle interior video captured by the in-camera, and generates video data displaying the frame of view of the driver and the action of the driver together with the vehicle exterior video based on a result of the action analysis (step S).

Specifically, as shown in, the teaching material video is generated in a manner of displaying a frame of view R, an object frame R, and an action display area Rtogether with a vehicle exterior video V by the out-camera

The frame of view Ris a detection frame indicating an actual field of view of the driver. The frame of view Ris superimposed and displayed on the vehicle exterior video V, so that the participant may clearly grasp the actual field of view of the driver. A position, a size, a shape, and the like of the frame of view Rare estimated by analyzing, by the server device, the vehicle interior video captured by the in-camerausing, for example, an artificial intelligence (AI) model for image recognition.

The server devicerecognizes the face of the driver or an object other than the face from the vehicle interior video using, for example, the AI model. Further, the server deviceextracts, for example, the feature points of the face from the recognized face. The server deviceestimates an orientation of the face of the driver, an eye open state of the driver and the like based on, for example, positions of the extracted feature points of the face. The eye open state of the driver means an opened and closed state of eyelids of the driver. Then, the server deviceestimates the frame of view Rbased on an estimation result.

The server deviceestimates a position of the frame of view Rdisplayed on the vehicle exterior video V according to, for example, the estimated orientation of the face of the driver. Accordingly, the server devicemay display the frame of view Rin a position close to the actual field of view according to the orientation of the face of the driver. The server deviceestimates the size of the frame of view Rdisplayed on the vehicle exterior video V according to, for example, the estimated eye open state of the driver. Accordingly, the server devicemay display the frame of view Rhaving a size close to the actual field of view according to the eye open state of the driver. In, the frame of view Ris shown in an elliptical shape, and a shape of the frame of view Ris not limited.

Further, when estimating that the estimated position of the frame of view Ris outside an imaging range of the vehicle exterior video V, for example, the server devicedoes not display the frame of view Ron the vehicle exterior video V. Further, when estimating that the eyes are closed based on the estimated eye open state, for example, the server devicedoes not display the frame of view Ron the vehicle exterior video V.

In this manner, the server devicemay generate a teaching material video in which display control of switching the display and non-display of the frame of view Ris executed according to the orientation of the face of the driver, the eye open state and the like. The participant may easily grasp the actual field of view of the driver by viewing the teaching material video in which the display control is executed such that the display and non-display of the frame of view Ris switched.

The object frame Ris a detection frame for an object in an accident, a near-miss and the like. The object frame Ris estimated by analyzing, by the server device, the vehicle exterior video V captured by the out-camerausing, for example, the above-described AI model.

The object frame Ris superimposed and displayed on the vehicle exterior video V, so that the participant may easily grasp the object in the accident, the near-miss and the like.

The action display area Ris a display area for an action (may also be read as “state”) of the driver. The action of the driver displayed in the action display area Ris estimated by the server devicebased on an object other than the face, the orientation of the face, the eye open state, and the like recognized from the vehicle interior video captured by the in-camera

The object other than the face is, for example, the hands of the driver or a smartphone. The action of the driver is, for example, “dozing”, “operating phone”, or “looking aside”. “Operating phone” corresponds to an example of an “operation while driving”, which involves operating an object such as a phone or a car navigation system during driving. The server devicegenerates the action display area Rin a manner of displaying the action of the driver in, for example, text information.shows an example in which “operating phone” is displayed in text information in the action display area R.

Since the action of the driver is displayed in text information in the action display area R, the participant may easily grasp the action of the driver during the accident, the near-miss and the like. Further, since the display control of displaying and not displaying the frame of view Ris executed together with the display of the action display area R, the participant may easily grasp the actual field of view of the driver and what led the driver to the field of view during the accident, the near-miss and the like.

As shown in, the frame of view Ris preferably subjected to image processing such that, for example, brightness and transparency in the frame of view Rare higher than those in a region other than the frame of view Rof the vehicle exterior video V. That is, the image processing is preferably executed such that visibility of the frame of view Ris higher than that of the other region of the vehicle exterior video V. Accordingly, the participant may clearly distinguish and grasp the actual field of view of the driver from the other region.

The action display area Rmay also display the action of the driver in a manner of processing a background to be transparent and directly superimposing the text information indicating the action of the driver on the vehicle exterior video V. In addition, the action display area Ris not limited to displaying the action of the driver in text information, and may also display the action of the driver in a prescribed icon (including a symbol, a mark, and the like) indicating the action of the driver.

Description is referred back to. Then, the server devicedelivers the generated teaching material video to the participating device(step S). When viewing the teaching material video delivered to the participating device, the participant grasps the actual field of view of the driver and the action of the driver leading to the field of view in a scene of a teaching material of safe driving guidance or the like.

In this manner, in the teaching material video generation method according to the embodiment, the server deviceacquires the vehicle interior video captured by the in-cameraand the vehicle exterior video captured by the out-camera, and executes the action analysis of the feature points of the face, the eyes, or the hand of the driver based on the vehicle interior video. Further, the server devicegenerates video data displaying the frame of view of the driver and the action of the driver together with the vehicle exterior video based on a result of the action analysis.

According to the teaching material video generation method according to the embodiment, the viewer of the vehicle exterior video V may grasp the actual field of view of the driver and what led the driver to the field of view.

Hereinafter, a configuration example of the teaching material video generation systemincluding the drive recorderto which the teaching material video generation method according to the above-described embodiment is applied will be described more specifically.

shows a configuration example of the teaching material video generation systemaccording to the embodiment. As shown in, the teaching material video generation systemincludes one or more drive recorders-,-, . . .-(m is a natural number), the server device, and one or more participating devices-,-, . . .-(n is a natural number).

The drive recorder, the server device, and the participating deviceare communicably connected via a network N such as the Internet, a mobile phone line network, or a cellular vehicle to everything (C-V2X) communication network.

As described above, the drive recorderrecords vehicle data, which includes vehicle interior and exterior video data captured by the in-cameraand the out-cameraand various types of data indicating a situation of the vehicle, in a circular buffer memory in an over-writable manner for a predetermined period.