Imaging device and operating method of the imaging device

This application is based on and claims priority under 35 U.S.C. § 119 to Japanese Patent Application No. 2024-013886, filed on Feb. 1, 2024, in the Japan Patent Office, the disclosure of which is incorporated by reference herein in its entirety.

Devices, apparatuses, and method consistent with the present disclosure relate to an imaging device and an operating method of the imaging device.

An in-cabin driver monitoring system is a system for monitoring the appearance of a driver in a vehicle. In this system, an image is generated under various conditions and, a driver's gaze, etc. are detected, thereby detecting the appearance of the driver, such as drowsy driving.

However, in certain circumstances, artifacts due to differences in the driver's position, etc. may occur in the image and, due to the occurrence of the artifacts, the detection accuracy of the driver's gaze, etc. is reduced.

It is an aspect to provide an imaging device, an imaging method, and an imaging program configured to obtain a high-resolution captured image without artifacts, in which the influence of background light is removed even when a subject moves in any direction.

According to an aspect of one or more embodiments, there is provided an imaging device comprising an imaging unit configured to capture at least three frame images exposed in synchronization with a turning-on and a turning-off of a light emitter that irradiates light to a subject, the at least three frame images including a turning-on frame image exposed when the light emitter is turned on, a turning-off frame image exposed when the light emitter is turned off, and another one of the turning-on frame image or the turning-off frame image; a motion vector calculator configured to calculate a motion vector of each pixel from the two of the turning-on frame images or the two of the turning-off frame images; a motion vector length adjuster configured to adjust a length of the motion vector of each pixel according to a time difference between a time at which the turning-on frame image is captured and at time at which the turning-off frame image is captured; a frame image corrector configured to generate a virtual frame image by correcting each pixel of the turning-off frame image based on the motion vector of each pixel after the motion vector is adjusted; and an image processor configured to subtract the virtual frame image from the turning-on frame image.

According to another aspect of one or more embodiments, there is provided an operating method of an imaging device, the operating method comprising capturing at least three frame images exposed in synchronization with a turning-on and a turning-off of a light emitter that irradiates light to a subject, the at least three frame images including a turning-on frame image exposed when the light emitter is turned on, a turning-off frame image exposed when the light emitter is turned off, and another one of the turning-on frame image or the turning-off frame image; calculating a motion vector of each pixel from the two of the turning-on frame images or the two of the turning-off frame images; adjusting a length of the motion vector of each pixel according to a time difference between a time at which the turning-on frame image is captured and a time at which the turning-off frame image is captured; correcting each pixel of the turning-off frame image based on the motion vector of each pixel after the length of the motion vector is adjusted to generate a virtual frame image; and subtracting the virtual frame image from the turning-on frame image.

According to yet another aspect of one or more embodiments, there is provided an imaging device comprising a memory storing one or more instructions; and a processor configured to access the memory and execute the one or more instructions stored in the memory to cause the processor to capture at least three frame images by performing exposure in synchronization with a turning-on and a turning-off of a light emitter that irradiates light to a subject, the at least three frame images including a turning-on frame image exposed when the light emitter is turned on, a turning-off frame image exposed when the light emitter is turned off, and another one of the turning-on frame image or the turning-off frame image; calculate a motion vector of each pixel from the two of the turning-on frame images or the two of the turning-off frame images; adjust a length of the motion vector of each pixel according to a time difference between a time at which the turning-on frame image is captured and a time at which the turning-off frame image is captured; correct each pixel of the turning-off frame image based on the motion vector of each pixel after the length of the motion vector is adjusted to generate a virtual frame image; and subtracting the virtual frame image from the turning-on frame image.

In an in-cabin driver monitoring system, in order to eliminate the influence of background light (external light) on an acquired infrared image, a differential image is generated between an image when an LED that irradiates a shooting area is turned on and an image when the LED is turned off. Then, based on the differential image, the driver's gaze, etc. are detected, thereby detecting the appearance of the driver, such as drowsy driving.

However, as described above, artifacts due to differences in the driver's position, etc. occur in the differential image when the driver or the camera moves. In addition, due to the occurrence of the artifacts, the detection accuracy of the driver's gaze, etc. is reduced.

In some situations, the differential image may be generated by a difference in brightness between a first image captured when the subject is not illuminated and a second image captured when the subject is illuminated. In the differential image, a pixel area composed of pixels having a brightness higher than a certain brightness value is extracted. Based on the extracted pixel area, an amount of movement of the subject between a shooting timing of the first image and the shooting timing of the second image may be estimated. Then, based on the estimated amount of movement, etc., a subject portion in one of the first image and the second image may be moved aside and then a differential image may be generated again. Thus, even when the subject moves somewhat, an appropriate subject image may be obtained, and the authentication system may be improved.

However, because the amount of movement of the subject is estimated from a differential image between an image captured when the subject is illuminated and an image captured when the subject is not illuminated, an accuracy of the estimation of the amount of movement, etc. is limited and a clear image with the influence of background light removed may not be obtained.

Various embodiments described herein address the disadvantages described above. It is thus an aspect to provide an imaging device, an imaging method, and an imaging program configured to obtain a high-resolution captured image without artifacts, in which the influence of background light is removed even when the subject moves in any direction.

Hereinafter, an imaging device and imaging program according to various embodiments will be described in detail based on the drawings. The embodiments are only examples, and various modifications may be made from the embodiments. Hereinafter, the same reference numerals in the drawings refer to the same components, and a size of each component in the drawings is expressed at a different ratio than an actual size of the component for the sake of clarity and convenience of explanation.

Hereinafter, the expression “upper” or “above” includes not only those elements or components that are directly above/below/left/right in contact, but also those elements or components that are above/below/left/right without contact. Terms, such as “first” and “second”, are used to describe various components, but are used only for the purpose of distinguishing one component from another. Thus, a “first” element, component, region, layer or section described below could be termed a “second” element, component, region, layer or section, without departing from the spirit and scope of the present disclosure. Such terms do not limit the material or structure of the components to be different.

A component expressed in the singular includes plural components unless the context clearly indicates otherwise. In addition, when a part in a specification is said to “comprise” or “include” a component, this description does not mean that the part excludes other components, but rather that the part may include other components, unless otherwise stated.

In addition, terms, such as “unit” and “module” described in the specification, mean a unit that processes one or more functions or operations, which are implemented by hardware or software, or by a combination of hardware and software.

Hereinafter, various embodiments are described.

1 FIG. 2 FIG. 3 FIG. 1 210 200 110 120 100 500 is a block diagram illustrating a hardware configuration of an imaging deviceaccording to an embodiment.is a block diagram illustrating a function of a controllerof an image processing deviceaccording to an embodiment.is a diagram illustrating an example of a positional relationship between a light emitterand a cameraof an imaging unitand a subject, according to an embodiment.

1 100 200 100 200 100 110 120 130 140 200 210 220 230 The imaging devicemay include the imaging unitand the image processing device. The imaging unitand the image processing deviceare connected to each other for communication. The imaging unitmay include a light emitter, a camera, a controller, and a communicator. The image processing devicemay include a controller, a memory, and a communicator.

110 500 110 130 500 110 110 110 The light emitterirradiates (or, illuminates) light to the subject. Specifically, the light emitteris turned on or off according to control by the controllerand irradiates light to the subjectwhen turned on. In some embodiments, the light emittermay be configured by, for example, a light-emitting diode (LED) that emits infrared rays. In some embodiments, a wavelength of light emitted by the LED may be 940 nm. The light emittermay include an LED that emits visible light. In some embodiments, the light emittermay be composed of Vicsel.

120 700 120 500 700 120 710 110 720 110 110 130 120 120 4 FIG. 5 FIG. The camerahas an imaging element including a plurality of elements (pixels) and may integrate light detected by the elements and output integrated light as a frame image(see) that is a digital signal. The cameracaptures an image of the subjectas the frame imagethat is temporally continuous. Specifically, the cameracaptures an exposure frame image (hereinafter, referred to as a turning-on state exposure frame image)exposed when the light emitteris turned on and an exposure frame image (hereinafter, referred to as a turning-off state exposure frame image)exposed when the light emitteris turned off by performing exposure in synchronization with the turning-on and turning-off of the light emitteraccording to the control by the controller(see, etc.). The cameramay be, for example, an infrared camera. The cameramay be a visible light camera.

130 130 100 130 110 110 130 120 120 120 110 The controllermay be configured by a central processing unit (CPU) and a memory. The controllercontrols each element constituting the imaging unitand performs various calculation processes. The controllertransmits a control signal to the light emitterto turn the light emitteron or off. The controllertransmits a control signal to the camerato expose the cameraat a certain timing, the camerabeing synchronized with the control signal for turning the light emitteron or off.

140 700 200 700 710 720 The communicatortransmits the frame imageto the image processing device. The frame imagemay include the turning-on state exposure frame imageand the turning-off state exposure frame image.

210 210 200 The controllermay be configured by a CPU and a memory. The controllercontrols each element constituting the image processing deviceand performs various calculation processes.

2 FIG. 210 211 212 213 214 215 210 211 212 213 214 215 As illustrated in, the controllerfunctions as a motion vector calculator, a motion vector length adjuster, a turning-off state exposure frame image corrector, a low-pass filter, and a turning-on state exposure frame image processoras the CPU executes a program. The controllermay include the motion vector calculator, the motion vector length adjuster, the turning-off state exposure frame image corrector, the low-pass filter, and the turning-on state exposure frame image processor.

210 720 721 710 210 710 500 110 500 5 FIG. 3 FIG. As described below, the controllerperforms a process (hereinafter, also simply referred to as “processing”) of subtracting a turning-off state exposure frame image(a virtual turning-off state exposure frame imagedescribed below (see, etc.)), in which the position of each pixel is corrected, from the turning-on state exposure frame image. Thus, the controllerremoves the influence of background light (external light) from the turning-on state exposure frame image. The subjectmay be irradiated with light from the light emitterand external light, such as sunlight (see). The subjectmay be, for example, a person, but may also be an animal or a moving object.

211 730 710 720 730 730 700 700 730 5 FIG. The motion vector calculatorcalculates a motion vector(see, etc.) of each pixel from two turning-on state exposure frame imagesor two turning-off state exposure frame images. For example, an optical flow algorithm may be used to calculate the motion vector. The motion vectoris a vector of which elements include a length and a direction of a line connecting each pixel of a temporally previous frame imageto each corresponding pixel of a temporally subsequent frame image. There are various methods of calculating the motion vectorother than the optical flow algorithm, and any of these methods may be used, and thus, descriptions thereof are omitted for conciseness.

212 730 710 720 700 730 710 720 710 730 710 710 212 730 730 710 720 700 730 700 100 The motion vector length adjusteradjusts the length of the motion vectorof each pixel according to a time difference between the turning-on state exposure frame imageof a processing target and the turning-off state exposure frame imageused for the corresponding processing (the above-described subtraction processing). One of two frame imagesused to calculate the motion vectormay be the turning-on state exposure frame imageof the processing target or the turning-off state exposure frame imageused for the corresponding processing. In the present embodiment, the two turning-on state exposure frame imagesare used to calculate the motion vector, and one of the two turning-on state exposure frame imagesis the turning-on state exposure frame imageof the processing target. The motion vector length adjustermay adjust the length of the motion vectorof each pixel by multiplying, by the length of the motion vector, the ratio of the time difference between the turning-on state exposure frame imageof the processing target and the turning-off state exposure frame imageused for the corresponding processing to the time difference between the two frame imagesused to calculate the motion vector. The time difference between the two frame imagesis calculated, for example, based on a frame rate of the imaging unit.

213 720 731 721 110 710 213 721 730 720 730 720 730 720 710 720 The turning-off state exposure frame image correctorcorrects each pixel of the turning-off state exposure frame imageused for the processing, based on the motion vectorof each pixel after the length adjustment, thereby calculating a virtual turning-off state exposure frame imagethat is estimated to be captured when the light emitteris turned off during exposure of the turning-on state exposure frame imageof the processing target. Specifically, the turning-off state exposure frame image correctorgenerates a virtual turning-off state exposure frame imageby shifting, by the length of the motion vectorafter length adjustment, each pixel of the turning-off state exposure frame imageused for the processing in a direction along the direction of the corresponding motion vector. The direction in which each pixel of the turning-off state exposure frame imageused for the processing is shifted is the same direction as or opposite to the corresponding motion vector. The direction in which each pixel of the turning-off state exposure frame imageused for the processing is shifted varies depending on the temporal positional relationship between the turning-on state exposure frame imageof the processing target and the turning-off state exposure frame imageused for the processing.

214 721 721 The low-pass filterfilters the virtual turning-off state exposure frame image, thereby reducing the influence of edge noise due to movement error in shift of each pixel when generating the virtual turning-off state exposure frame image. For example, a Gaussian filter, a median filter, or a bilateral filter may be used as the low-pass filter.

215 721 710 The turning-on state exposure frame image processorperforms the processing of subtracting the virtual turning-off state exposure frame imagefrom the turning-on state exposure frame imageof the processing target.

1 220 210 In an embodiment, the imaging devicemay include a memory and a processor. The memory may be the memory. The processor may be the controller. The memory may store one or more instructions. The processor may execute one or more instructions stored in the memory. The processor may perform an imaging method by executing the one or more instructions.

For example, the processor may execute the one or more instructions to capture a turning-on state exposure frame image exposed when a light emitter is turned on and a turning-off state exposure frame image exposed when the light emitter is turned off by performing exposure in synchronization with the turning-on and turning-off of the light emitter that irradiates light to the subject, to calculate a motion vector of each pixel from two turning-on state exposure frame images or two turning-off state exposure frame images, to adjust the length of the motion vector of each pixel according to the time difference between the turning-on state exposure frame image of a processing target and the turning-off state exposure frame image used for processing, to correct each pixel of the turning-off state exposure frame image used for the processing based on the motion vector of each pixel after the length adjustment, thereby calculating a virtual turning-off state exposure frame image that is estimated to be captured when the light emitter is turned off during exposure of the turning-on state exposure frame image of the processing target, and to subtract the calculated virtual turning-off state exposure frame image from the turning-on state exposure frame image of the processing target.

4 FIG. 700 730 720 is a diagram for explaining an example of two frame imagesused to calculate a motion vectorand a turning-off state exposure frame imageused for processing, according to an embodiment.

130 110 130 120 An illumination light control signal is a control signal for the controllerto turn the light emitteron or off. An imaging element frame signal is a control signal for the controllerto expose the cameraand read an image signal from each element (pixel).

700 710 700 110 720 700 710 720 700 730 4 FIG. A first frame image, a second frame image, and a third frame image are frame imagesthat are sequentially captured in time. In the example of, the first frame image and the third frame image are turning-on state exposure frame imagesand are frame imagescaptured by illumination light irradiated by the light emitterand background light (including reflected light). The second frame image is a turning-off state exposure frame imageand is a frame imagecaptured only by background light (including reflected light) without irradiating illumination light. When the first frame image is the turning-on state exposure frame imageof the processing target, the second frame image may be the turning-off state exposure frame imageused for the processing, and the first frame image and the third frame image may be two frame imagesused to calculate the motion vector.

5 FIG. 730 730 721 721 710 is a diagram for explaining an example of calculating a motion vector, adjusting the length of the motion vector, calculating a virtual turning-off state exposure frame image, and subtracting the virtual turning-off state exposure frame imagefrom a turning-on state exposure frame image, according to an embodiment.

1 2 3 710 720 710 700 710 720 700 730 730 2 3 500 1 5 FIG. 5 FIG. 5 FIG. 5 FIG. 5 FIG. 5 FIG. 5 FIG. In an embodiment, the first frame image (() of), the second frame image (() of), and the third frame image (() of) are a turning-on state exposure frame image, a turning-off state exposure frame image, and a turning-on state exposure frame image, respectively. These frame imagesare captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first turning-on state exposure frame image, a first turning-off state exposure frame image, and a second turning-on state exposure frame image, respectively. The first frame image is the turning-on state exposure frame imageof a processing target. The second frame image is the turning-off state exposure frame imageused for the processing. The first frame image and the third frame image are two frame imagesused to calculate the motion vector. The second frame image used for the processing is interpolated into the first frame image and the third frame image that are used to calculate the motion vector. In, in the diagram of the second frame image (() of) and the diagram of the third frame image (() of), the outline of the subjectin the first frame image (() of) is indicated by a broken line for easy explanation.

730 4 730 730 4 4 730 5 FIG. 5 FIG. 5 FIG. The motion vectoris calculated for each pixel from the first frame image and the third frame image (see () of). For example, the motion vectorfor a given pixel indicates, as a vector, a change of location of the given pixel from the first frame image to the third frame image. The motion vectoris indicated by an arrow in () of. In () of, the motion vectoris also shown in an enlarged view surrounded by a circle.

730 The length of the calculated motion vectoris adjusted according to the ratio of the time difference between the first frame image and the second frame image to the time difference between the first frame image and the third frame image.

721 730 731 5 5 5 731 5 FIG. 5 FIG. 5 FIG. A virtual turning-off state exposure frame imageis calculated (e.g., made or generated) by performing a correction to shift each pixel of the second frame image in the opposite direction to the motion vectorby the length of the motion vectorafter the length adjustment (see () of). The shift direction and shift amount of each pixel in the correction are indicated by arrows in () of. In () of, the motion vectorafter the length adjustment is also shown in an enlarged view surrounded by a circle.

721 721 6 5 FIG. In order to reduce the influence of edge noise due to a movement error in a shift of each pixel when calculating the virtual turning-off state exposure frame image, a filter may be applied to the virtual turning-off state exposure frame image(see () of).

721 7 5 FIG. A process of subtracting the virtual turning-off state exposure frame imagefrom the first frame image is performed (see () of).

6 FIG. 1 FIG. 1 is a flowchart illustrating an example of operations of the imaging deviceof, according to an embodiment.

101 105 130 100 210 200 130 130 220 130 101 210 220 210 102 105 The operations S-Sof the flowchart may be executed, according to a program, by the cooperative operation of the controllerof the imaging unitand the controllerof the image processing device. In some embodiments, the controllermay access a first program stored in a memory of the controlleror stored in the memory, and execute the first program to cause the controllerto perform operation S. In some embodiments, the controllermay access a second program stored in the memoryand execute the second program to cause the controllerto perform operations S-S.

130 120 110 101 1 130 200 140 The controllercaptures the first frame image, the second frame image, and the third frame image, by the camera, in synchronization with the turning-on, turning-off, and turning-on of the light emitter(S). In an embodiment, the imaging devicemay generate a first frame image, a second frame image, and a third frame image. The controllertransmits the first frame image, the second frame image, and the third frame image to the image processing devicethrough the communicator.

210 730 102 210 230 730 210 730 The controllercalculates a motion vectorfrom the first frame image and the third frame image (S). For example, in an embodiment, the controllerreceives the first frame image, the second frame image, and the third frame image through the communicatorand calculates the motion vectorfrom the first frame image and the third frame image. In an embodiment, the controllermay calculate the motion vectorbased on the first frame image and the third frame image.

210 730 103 210 730 210 730 The controlleradjusts the length of the motion vectoraccording to the ratio of the time difference between the first frame image and the second frame image to the time difference between the first frame image and the third frame image (S). In an embodiment, the controllermay adjust the length of the motion vectorbased on a ratio of a first time difference between the first frame image and the second frame image to a second time difference between the first frame image and the third frame image. In an embodiment, the controllermay adjust the length of the motion vectorbased on the first time difference between a time at which the first frame image is captured and a time at which the third frame image is captured and the second time difference between the time at which the first frame image is captured and a time at which the second frame image is captured.

210 730 731 721 104 210 721 730 730 210 730 721 210 730 721 The controllershifts each pixel of the second frame image in the opposite direction to the motion vectorby the length of the motion vectorafter the length adjustment, to calculate (or generate) a virtual turning-off state exposure frame image(S). In an embodiment, the controllermay calculate the virtual turning-off state exposure frame imagebased on the motion vectorand the adjusted length of the motion vector. For example, the controllermay shift each pixel of the second frame image in the opposite direction to the motion vectorto calculate the virtual turning-off state exposure frame image. For example, the controllermay shift each pixel of the second frame image by the adjusted length of the motion vectorto calculate the virtual turning-off state exposure frame image.

210 721 105 The controllerperforms a process of subtracting the virtual turning-off state exposure frame imagefrom the first frame image (S).

210 710 710 6 FIG. The controllermay perform the process of the flowchart illustrated inby sequentially processing turning-on state exposure frame images, which are sequentially received, as turning-on state exposure frame imagesof a processing target.

1 6 FIGS.- 720 710 500 120 According to the embodiment illustrated with respect to, because the turning-off state exposure frame imagesubtracted from the turning-on state exposure frame imageis shifted by a pixel unit, an image without artifacts, in which the influence of background light is removed even when the subjectmoves in any direction and even when the cameramoves, may be obtained.

1 6 FIGS.- 1 6 FIGS.- 730 730 730 730 In the embodiment illustrated with respect to, the length of the motion vectoris adjusted for each pixel. However, this embodiment is only an example. In an embodiment, an average length of the motion vectorof each pixel is adjusted, and the motion vectorof each pixel except for a certain pixel of which the length of the motion vectoris less than or equal to a threshold value is uniformly adjusted to the average after the adjustment. In other respects, the embodiment is the same as the embodiment illustrated with respect to, and thus, redundant descriptions are omitted or simplified for conciseness.

212 730 730 730 730 730 2 FIG. The motion vector length adjusterillustrated insets a pixel, of which the length of the motion vectoris less than or equal to a threshold value, as a certain pixel of which the length of the motion vectoris not adjusted. For example, the certain pixel may be a pixel of which the length of the motion vectoris less than or equal to the threshold value. The certain pixel may be a pixel of which the length of the motion vectoris not adjusted. The certain pixel may be excluded from the adjustment of the length of the motion vector.

730 730 For example, a first pixel may refer to a pixel of which the length of the motion vectoris less than or equal to the threshold value. That is, the first pixel may be the certain pixel. A second pixel may refer to a pixel of which the length of the motion vectoris greater than the threshold value. That is, the second pixel may be a pixel other than the certain pixel.

212 730 212 730 710 720 720 730 212 730 The motion vector length adjustercalculates the average of the motion vectorof each pixel. The motion vector length adjusteradjusts an average length of the motion vectoraccording to the time difference between a time at which the turning-on state exposure frame imageof the processing target is captured and a time at which the turning-off state exposure frame imageused for the processing is captured, and corrects each pixel (for example, the second pixel) other than the certain pixel of the turning-off state exposure frame imageused for the processing, based on the average of the motion vectorof each pixel after the adjustment. The motion vector length adjustermay correct the second pixel based on the average of the motion vectorof each pixel after the adjustment.

730 730 730 730 710 The correction of each pixel other than the certain pixel may be performed by generating a vector mask, in which a motion vectorhaving a length less than or equal to a certain threshold value is set to 0 and a motion vectorother than the motion vectorhaving the length less than or equal to the certain threshold value is set to 1, and using the vector mask. That is, only for the pixel of ‘1’ in the vector mask, correction may be made based on the average of the motion vectorof each pixel after the adjustment. The threshold value may be set to an appropriate value through experiment from a viewpoint of an image quality of the turning-on state exposure frame imageafter the processing and/or a viewpoint of a reduction of an amount of calculation. For example, the threshold value may be determined in advance.

7 FIG. 730 730 721 721 710 is a diagram for explaining an example of calculating a motion vector, adjusting the length of the motion vector, calculating a virtual turning-off state exposure frame image, and subtracting the virtual turning-off state exposure frame imagefrom a turning-on state exposure frame image, according to an embodiment.

1 2 3 710 720 710 700 710 720 700 730 730 2 3 500 1 7 FIG. 7 FIG. 7 FIG. 7 FIG. 7 FIG. 7 FIG. 7 FIG. The first frame image (() of), the second frame image (() of), and the third frame image (() of) are a turning-on state exposure frame image, a turning-off state exposure frame image, and a turning-on state exposure frame image, respectively. These frame imagesare captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first turning-on state exposure frame image, a first turning-off state exposure frame image, and a second turning-on state exposure frame image, respectively. The first frame image is the turning-on state exposure frame imageof a processing target. The second frame image is the turning-off state exposure frame imageused for the processing. The first frame image and the third frame image are two frame imagesused to calculate the motion vector. The second frame image used for the processing is interpolated into the first frame image and the third frame image that are used to calculate the motion vector. In, in the diagram of the second frame image (() of) and the diagram of the third frame image (() of), the outline of the subjectin the first frame image (() of) is indicated by a broken line.

730 4 730 4 7 FIG. 7 FIG. The motion vectoris calculated for each pixel from the first frame image and the third frame image (see () of). The motion vectoris indicated by an arrow in () of.

730 5 730 730 5 5 7 FIG. 7 FIG. 7 FIG. A vector mask, in which a motion vectorhaving a length less than or equal to a threshold value is set to 0 (a black part of the drawing in () of) and a motion vectorother than the motion vectorhaving the length less than or equal to the threshold value is set to 1 (a white part of the drawing in () of), is generated (or made) (see () of).

730 An average of the calculated motion vectoris calculated, and the average is adjusted according to the ratio of the first time difference between a time at which the first frame image is captured and a time at which the second frame image is captured to the second time difference between the time at which the first frame image is captured and a time at which the third frame image is captured.

721 730 731 6 7 FIG. A virtual turning-off state exposure frame imageis calculated (e.g., made or generated) by performing, by using the vector mask, a correction to shift each pixel other than a certain pixel of the second frame image in the opposite direction to the average of the motion vectorafter the adjustment by the length of the average of the motion vectorafter the adjustment (see () of).

721 721 7 721 7 FIG. In order to reduce the influence of edge noise due to movement error in shift of each pixel when calculating the virtual turning-off state exposure frame image, a filter may be applied to the virtual turning-off state exposure frame image(see () of). That is, the virtual turning-off state exposure frame imagemay be filtered with a filter.

721 8 7 FIG. A process of subtracting the virtual turning-off state exposure frame imagefrom the first frame image is performed (see () of).

8 FIG. 1 FIG. 1 is a flowchart illustrating an example of operations of the imaging deviceof, according to an embodiment.

201 206 130 100 210 200 130 130 220 130 201 210 220 210 202 206 The operations Sto Sof the flowchart may be executed, according to a program, by the cooperative operation of the controllerof the imaging unitand the controllerof the image processing device. In some embodiments, the controllermay access a first program stored in a memory of the controlleror stored in the memory, and execute the first program to cause the controllerto perform operation S. In some embodiments, the controllermay access a second program stored in the memoryand execute the second program to cause the controllerto perform operations S-S.

130 120 110 201 130 200 140 The controllercaptures the first frame image, the second frame image, and the third frame image, by the camera, in synchronization with the turning-on, turning-off, and turning-on of the light emitter(S). The controllertransmits a first frame image, a second frame image, and a third frame image to the image processing devicethrough the communicator.

210 730 202 210 230 730 210 730 The controllercalculates a motion vectorfrom the first frame image and the third frame image (S). For example, the controllerreceives the first frame image, the second frame image, and the third frame image through the communicatorand calculates the motion vectorfrom the first frame image and the third frame image. The controlleralso calculates the average of the motion vector.

210 730 203 210 730 203 210 730 730 730 The controllergenerates a vector mask based on the length of the motion vector(S). For example, the controllergenerates the vector mask based on the length of the motion vectorof each pixel (S). That is, the controllergenerates (or makes) a vector mask, in which a motion vectorof a certain pixel having a length less than or equal to a threshold value is set to 0 and a motion vectorother than the motion vectorhaving the length less than or equal to the threshold value is set to 1.

210 730 204 The controlleradjusts an average length of the motion vectoraccording to the ratio of the time difference between a time at which the first frame image is captured and a time at which the second frame image is captured to the time difference between the time at which the first frame image is captured and a time at which the third frame image is captured (S).

210 721 731 730 205 The controllercalculates a virtual turning-off state exposure frame imageby shifting each pixel other than a certain pixel of the second frame image by the average length of the motion vectorafter the length adjustment in the opposite direction to the average of the motion vectorby using a vector mask (S).

210 721 206 The controllerperforms a process of subtracting the virtual turning-off state exposure frame imagefrom the first frame image (S).

7 8 FIGS.- 730 According to the embodiment illustrated in, by uniformly shifting each pixel other than a certain pixel based on the average of the length of the motion vector, the amount of calculation used for processing may be effectively reduced.

1 6 FIGS.- 9 FIG. 1 6 FIGS.- 730 715 710 730 730 715 In the embodiment illustrated with respect to, the motion vectoris calculated for all pixels. In an embodiment, a face region(see) is detected from a turning-on state exposure frame imagethat is used to calculate the motion vector, and the motion vectoris calculated only for each pixel within the detected face region. In other respects, the embodiment is the same as the embodiment illustrated with respect to, and thus, redundant descriptions are omitted or simplified for conciseness.

211 715 710 730 211 730 715 2 FIG. The motion vector calculatorillustrated indetects the face regionfrom the turning-on state exposure frame imagethat is used to calculate the motion vector. The motion vector calculatorcalculates the motion vectorof each pixel within the detected face region.

9 FIG. 730 730 721 721 710 is a diagram for explaining an example of calculating a motion vector, adjusting the length of the motion vector, calculating a virtual turning-off state exposure frame image, and subtracting the virtual turning-off state exposure frame imagefrom a turning-on state exposure frame image, according to an embodiment.

1 2 3 710 720 710 700 710 720 700 730 730 2 3 500 9 FIG. 9 FIG. 9 FIG. 9 FIG. 9 FIG. 9 FIG. The first frame image (() of), the second frame image (() of), and the third frame image (() of) are a turning-on state exposure frame image, a turning-off state exposure frame image, and a turning-on state exposure frame image, respectively. These frame imagesare captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first turning-on state exposure frame image, a first turning-off state exposure frame image, and a second turning-on state exposure frame image, respectively. The first frame image is the turning-on state exposure frame imageof a processing target. The second frame image is the turning-off state exposure frame imageused for the processing. The first frame image and the third frame image are two frame imagesused to calculate the motion vector. The second frame image used for the processing is interpolated into the first frame image and the third frame image that are used to calculate the motion vector. In, in the diagram of the second frame image (() of) and the diagram of the third frame image (() of), the outline of the subjectin the first frame image is indicated by a broken line.

715 500 4 715 715 715 9 FIG. A face region, which is a part including the face of the subject, is detected from the first frame image and set as a Region of Interest (ROI) (see () of). The detection of the face regionmay be performed using various methods. For example, the face regionmay be detected using a model of a neural network learned to detect the face regionfrom an image.

730 715 5 730 5 9 FIG. 9 FIG. The motion vectorof the face regionis calculated for each pixel from the first frame image and the third frame image (see () of). The motion vectoris indicated by an arrow in () of.

730 The length of the calculated motion vectoris adjusted according to the ratio of the time difference between the first frame image and the second frame image to the time difference between the first frame image and the third frame image.

721 715 730 731 6 6 9 FIG. 9 FIG. A virtual turning-off state exposure frame imageis calculated (e.g., generated or made) by performing a correction to shift each pixel of the face regionof the second frame image in the opposite direction to the motion vectorby the length of the motion vectorafter the length adjustment (see () of). The shift direction and shift amount of each pixel in the correction are indicated by arrows in () of.

721 721 7 9 FIG. In order to reduce the influence of edge noise due to movement error in shift of each pixel when calculating the virtual turning-off state exposure frame image, a filter may be applied to the virtual turning-off state exposure frame image(see () of).

721 8 9 FIG. A process of subtracting the virtual turning-off state exposure frame imagefrom the first frame image is performed (see () of).

10 FIG. 1 FIG. 1 is a flowchart illustrating an example of operations of the imaging deviceof, according to some embodiments.

301 306 130 100 210 200 130 130 220 130 301 210 220 210 302 306 The operations Sto Sof the flowchart may be executed, according to a program, by the cooperative operation of the controllerof the imaging unitand the controllerof the image processing device. In some embodiments, the controllermay access a first program stored in a memory of the controlleror stored in the memory, and execute the first program to cause the controllerto perform operation S. In some embodiments, the controllermay access a second program stored in the memoryand execute the second program to cause the controllerto perform operations S-S.

130 120 110 301 130 200 140 The controllercaptures the first frame image, the second frame image, and the third frame image, by the camera, in synchronization with the turning-on, turning-off, and turning-on of the light emitter(S). The controllertransmits the first frame image, the second frame image, and the third frame image to the image processing devicethrough the communicator.

210 715 302 210 230 715 210 715 The controllercalculates the face regionin the first frame image (S). For example, the controllerreceives the first frame image, the second frame image, and the third frame image through the communicatorand calculates the face regionin the first frame image. In an embodiment, the controllermay detect the face regionin the first frame image.

210 730 715 303 The controllercalculates the motion vectorof the face regionfrom the first frame image and the third frame image (S).

210 730 304 The controlleradjusts the length of the motion vectoraccording to the ratio of the time difference between a time at which the first frame image is captured and a time at which the second frame image is captured to the time difference between the time at which the first frame image is captured and a time at which the third frame image is captured (S).

210 721 715 731 730 305 721 715 730 731 730 The controllercalculates a virtual turning-off state exposure frame imageby shifting each pixel of the face regionof the second frame image by the length of the motion vectorafter the length adjustment in the opposite direction to the motion vector(S). Thus, a virtual turning-off state exposure frame image, in which each pixel of the face regionof the second frame image, from which the motion vectoris calculated, is shifted by the length of the motion vectorafter the length adjustment in the opposite direction to the motion vector, is calculated.

210 721 306 The controllerperforms a process of subtracting the virtual turning-off state exposure frame imagefrom the first frame image (S).

9 10 FIGS.- 730 721 715 720 710 500 According to the embodiment illustrated with respect to, because a target pixel, from which the motion vectoris calculated, and a target pixel for shift in calculating the virtual turning-off state exposure frame imageare limited to the face region, the amount of calculation may be suppressed. In addition, because the turning-off state exposure frame imagesubtracted from the turning-on state exposure frame imageis shifted by pixel unit, an image without artifacts, in which the influence of background light is removed regardless of the direction in which the subjectmoves, may be obtained.

1 6 FIGS.- 1 6 FIGS.- 721 721 721 721 In the embodiment illustrated with respect to, the virtual turning-off state exposure frame imageis filtered using one low-pass filter. In an embodiment, the virtual turning-off state exposure frame imagemay be filtered using multiple types of low-pass filters, and a process is performed using the virtual turning-off state exposure frame imageafter the filtering, which has the smallest standard deviation of the difference (or differential) from the virtual turning-off state exposure frame imagebefore the filtering. In other respects, the embodiment is the same as the embodiment illustrated with respect to, and thus, redundant descriptions are omitted or simplified for conciseness.

214 721 721 2 FIG. The low-pass filteroffilters the calculated virtual turning-off state exposure frame imageby using multiple types of low-pass filters, thereby generating a plurality of virtual turning-off state exposure frame imagesafter the filtering. For example, a Gaussian filter, a median filter, and/or a bilateral filter may be used as the multiple types of low-pass filters. Low-pass filters with different kernel sizes may be used as the multiple types of low-pass filters.

215 721 721 721 215 721 710 The turning-on state exposure frame image processorspecifies the virtual turning-off state exposure frame imageafter the filtering, which has the smallest standard deviation of the difference between the calculated virtual turning-off state exposure frame imageafter the filtering and the virtual turning-off state exposure frame imagebefore the filtering. The turning-on state exposure frame image processorperforms processing to subtract a certain virtual turning-off state exposure frame imagefrom the turning-on state exposure frame imageof a processing target.

11 FIG. 730 730 721 721 710 is a diagram for explaining an example of calculating a motion vector, adjusting the length of the motion vector, calculating a virtual turning-off state exposure frame image, and subtracting the virtual turning-off state exposure frame imagefrom a turning-on state exposure frame image, according to an embodiment.

1 2 3 710 720 710 700 710 720 700 730 730 2 3 500 1 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. 11 FIG. The first frame image (() of), the second frame image (() of), and the third frame image (() of) are a turning-on state exposure frame image, a turning-off state exposure frame image, and a turning-on state exposure frame image, respectively. These frame imagesare captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first turning-on state exposure frame image, a first turning-off state exposure frame image, and a second turning-on state exposure frame image, respectively. The first frame image is the turning-on state exposure frame imageof a processing target. The second frame image is the turning-off state exposure frame imageused for the processing. The first frame image and the third frame image are two frame imagesused to calculate the motion vector. The second frame image used for the processing is interpolated into the first frame image and the third frame image that are used to calculate the motion vector. In, in the diagram of the second frame image (() of) and the diagram of the third frame image (() of), the outline of the subjectin the first frame image (() in) is indicated by a broken line.

730 4 730 4 11 FIG. 11 FIG. The motion vectoris calculated for each pixel from the first frame image and the third frame image (see () of). The motion vectoris indicated by an arrow in () of.

730 The length of the calculated motion vectoris adjusted according to the ratio of the time difference between a time at which the first frame image is captured and a time at which the second frame image is captured to the time difference between the time at which the first frame image is captured and a time at which the third frame image is captured.

721 730 731 5 5 5 FIG. 11 FIG. A virtual turning-off state exposure frame imageis calculated (e.g., generated or made) by performing a correction to shift each pixel of the second frame image in the opposite direction to the motion vectorby the length of the motion vectorafter the length adjustment (see () of). The shift direction and shift amount of each pixel in the correction are indicated by arrows in () of.

721 721 6 11 FIG. By filtering the virtual turning-off state exposure frame imageby using multiple types of low-pass filters, a virtual turning-off state exposure frame imageafter the filtering is calculated (see () of).

721 721 721 721 7 11 FIG. The standard deviation of the difference between the virtual turning-off state exposure frame imagebefore the filtering and the virtual turning-off state exposure frame imageafter the filtering is calculated, and a virtual turning-off state exposure frame imagehaving the smallest calculated standard deviation is selected as a virtual turning-off state exposure frame imageused for the processing (see () of).

721 8 11 FIG. A process of subtracting the selected virtual turning-off state exposure frame imagefrom the first frame image is performed (see () of).

12 FIG. 1 FIG. 1 is a flowchart illustrating an example of operations of the imaging deviceof, according to an embodiment.

401 406 130 100 210 200 130 130 220 130 401 210 220 210 402 406 The operations S-Sof the flowchart may be executed, according to a program, by the cooperative operation of the controllerof the imaging unitand the controllerof the image processing device. In some embodiments, the controllermay access a first program stored in a memory of the controlleror stored in the memory, and execute the first program to cause the controllerto perform operation S. In some embodiments, the controllermay access a second program stored in the memoryand execute the second program to cause the controllerto perform operations S-S.

401 404 101 104 6 FIG. Because operations Sto Sare the same as operations Sto Sof, descriptions thereof are omitted for conciseness.

210 721 721 721 210 721 721 405 The controllerfilters the calculated virtual turning-off state exposure frame imageby using a plurality of low-pass filters and calculates the difference between the virtual turning-off state exposure frame imagebefore the filtering and the virtual turning-off state exposure frame imageafter the filtering. The controllerselects a virtual turning-off state exposure frame imagehaving the smallest standard deviation of a corresponding differential (or difference) as a virtual turning-off state exposure frame imageused for the processing (S).

210 721 210 721 210 721 721 721 210 721 721 721 721 210 721 721 For example, the controllermay filter the virtual turning-off state exposure frame imageby using a plurality of low-pass filters. The controllermay obtain (or generate) a plurality of filtered virtual turning-off state exposure frame images. The controllermay calculate the difference between the virtual turning-off state exposure frame imagebefore the filtering and the virtual turning-off state exposure frame imageafter the filtering with respect to the plurality of filtered virtual turning-off state exposure frame images. The controllermay select a virtual turning-off state exposure frame imagehaving the smallest standard deviation of the difference between the virtual turning-off state exposure frame imagebefore the filtering and the virtual turning-off state exposure frame imageafter the filtering from among the plurality of filtered virtual turning-off state exposure frame images. That is, the controllermay select a virtual turning-off state exposure frame imagehaving the smallest standard deviation of the difference from among the plurality of filtered virtual turning-off state exposure frame images.

210 721 406 The controllerperforms a process of subtracting the selected virtual turning-off state exposure frame imagefrom the first frame image (S).

11 12 FIGS.- 721 721 According to the embodiment illustrated with respect to, by performing the processing by using a virtual turning-off state exposure frame imagein which optimal filtering has been performed, edge noise due to the shift amount error of a pixel in the calculation of the virtual turning-off state exposure frame imagemay be further reduced.

1 6 FIGS.- 1 6 FIGS.- 710 730 710 710 730 710 In the embodiment illustrated with respect to, among two turning-on state exposure frame imagesused to calculate the motion vector, a turning-on state exposure frame imagecaptured first is determined as a processing target. In an embodiment, among two turning-on state exposure frame imagesused to calculate the motion vector, a turning-on state exposure frame imagecaptured later may be determined as a processing target. In other respects, the embodiment is the same as the embodiment illustrated with respect to, and thus, redundant descriptions are omitted or simplified for conciseness.

13 FIG. 730 730 721 721 710 is a diagram for explaining an example of calculating a motion vector, adjusting the length of the motion vector, calculating a virtual turning-off state exposure frame image, and subtracting the virtual turning-off state exposure frame imagefrom a turning-on state exposure frame image, according to an embodiment.

1 2 3 710 720 710 700 710 720 700 730 730 2 3 500 1 13 FIG. 13 FIG. 13 FIG. 13 FIG. 13 FIG. 13 FIG. 13 FIG. The first frame image (() of), the second frame image (() of), and the third frame image (() of) are a turning-on state exposure frame image, a turning-off state exposure frame image, and a turning-on state exposure frame image, respectively. These frame imagesare captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first turning-on state exposure frame image, a first turning-off state exposure frame image, and a second turning-on state exposure frame image, respectively. The third frame image is the turning-on state exposure frame imageof a processing target. The second frame image is the turning-off state exposure frame imageused for the processing. The first frame image and the third frame image are two frame imagesused to calculate the motion vector. The second frame image used for the processing is interpolated into the first frame image and the third frame image that are used to calculate the motion vector. In, in the diagram of the second frame image (() of) and the diagram of the third frame image (() of), the outline of the subjectin the first frame image (() of) is indicated by a broken line.

730 4 730 4 13 FIG. 13 FIG. The motion vectoris calculated for each pixel from the first frame image and the third frame image (see () of). The motion vectoris indicated by an arrow in () of.

730 The length of the calculated motion vectoris adjusted according to the ratio of the time difference between the second frame image and the third frame image to the time difference between the first frame image and the third frame image.

721 730 731 5 5 13 FIG. 13 FIG. A virtual turning-off state exposure frame imageis calculated by performing a correction to shift each pixel of the second frame image in the same direction as the motion vectorby the length of the motion vectorafter the length adjustment (see () of). The shift direction and shift amount of each pixel in the correction are indicated by arrows in the diagram of () of.

721 721 6 13 FIG. In order to reduce the influence of edge noise due to movement error in shift of each pixel when calculating the virtual turning-off state exposure frame image, a filter may be applied to the virtual turning-off state exposure frame image(see () of).

721 7 13 FIG. A process of subtracting the virtual turning-off state exposure frame imagefrom the third frame image is performed (see () of).

14 FIG. 1 FIG. 1 is a flowchart illustrating an example of operations of the imaging deviceof, according to an embodiment.

501 506 130 100 210 200 130 130 220 130 501 210 220 210 502 506 The operations Sto Sof the flowchart may be executed, according to a program, by the cooperative operation of the controllerof the imaging unitand the controllerof the image processing device. In some embodiments, the controllermay access a first program stored in a memory of the controlleror stored in the memory, and execute the first program to cause the controllerto perform operation S. In some embodiments, the controllermay access a second program stored in the memoryand execute the second program to cause the controllerto perform operations S-S.

130 120 110 501 130 200 140 The controllercaptures the first frame image, the second frame image, and the third frame image, by the camera, in synchronization with the turning-on, turning-off, and turning-on of the light emitter(S). The controllertransmits the first frame image, the second frame image, and the third frame image to the image processing devicethrough the communicator.

210 730 302 210 230 730 The controllercalculates the motion vectorfrom the first frame image and the third frame image (S). For example, the controllerreceives the first frame image, the second frame image, and the third frame image through the communicatorand calculates the motion vectorfrom the first frame image and the third frame image.

210 730 503 The controlleradjusts the length of the motion vector () according to the ratio of the time difference between the second frame image and the third frame image to the time difference between the first frame image and the third frame image (S).

210 730 731 721 504 The controllershifts each pixel of the second frame image in the same direction as the motion vectorby the length of the motion vectorafter the length adjustment and calculates a virtual turning-off state exposure frame image(S).

210 721 505 The controllerperforms a process of subtracting the virtual turning-off state exposure frame imagefrom the third frame image (S).

13 14 FIGS.- 13 14 FIGS.- 13 14 FIGS.- 1 6 FIGS.- 1 6 FIGS.- 720 710 500 120 721 720 720 710 710 720 710 720 721 720 720 710 710 720 710 720 According to the embodiment illustrated with respect to, because the turning-off state exposure frame imagesubtracted from the turning-on state exposure frame imageis shifted by pixel unit, an image without artifacts, in which the influence of background light is removed even when the subjectmoves in any direction and even when the cameramoves, may be obtained. In addition, in the embodiment illustrated with respect to, in the generation of the virtual turning-off state exposure frame image, each pixel of the turning-off state exposure frame imageis shifted in order to advance a turning-off state exposure frame imagecaptured first to a time point of exposure of a turning-on state exposure frame imagecaptured later. The embodiment illustrated with respect tomay be suitably employed in a case where the turning-on state exposure frame imageof a processing target is captured later than the turning-off state exposure frame imageused for processing and the capturing times of the two imagesandare relatively similar to each other. In the embodiment illustrated with respect to, in the generation of the virtual turning-off state exposure frame image, each pixel of the turning-off state exposure frame imageis shifted in order to return a turning-off state exposure frame imagecaptured later to a time point of exposure of a turning-on state exposure frame imagecaptured first. The embodiment illustrated with respect tomay be suitably employed in a case where the turning-on state exposure frame imageof a processing target is captured earlier than the turning-off state exposure frame imageused for processing and the capturing times of the two imagesandare relatively similar to each other.

1 6 FIGS.- 1 6 FIGS.- 710 730 720 730 In the embodiment illustrated with respect to, two turning-on state exposure frame imagesare used to calculate the motion vector. In an embodiment, two turning-off state exposure frame imagesare used to calculate the motion vector. In other respects, the embodiment is the same as the embodiment illustrated with respect to, and thus, redundant descriptions are omitted or simplified for conciseness.

15 FIG. 730 730 721 721 710 is a diagram for explaining an example of calculating a motion vector, adjusting the length of the motion vector, calculating a virtual turning-off state exposure frame image, and subtracting the virtual turning-off state exposure frame imagefrom a turning-on state exposure frame image, according to an embodiment.

1 2 3 720 710 720 700 710 720 700 730 730 2 3 500 1 15 FIG. 15 FIG. 13 FIG. 15 FIG. 15 FIG. 15 FIG. 15 FIG. The first frame image (() of), the second frame image (() of), and the third frame image (() of) are a turning-off state exposure frame image, a turning-on state exposure frame image, and a turning-off state exposure frame image, respectively. These frame imagesare captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first turning-off state exposure frame image, a first turning-on state exposure frame image, and a second turning-off state exposure frame image, respectively. The second frame image is the turning-on state exposure frame imageof a processing target. The first frame image is the turning-off state exposure frame imageused for the processing. The first frame image and the third frame image are two frame imagesused to calculate a motion vector. The second frame image, which is a processing target, is interpolated into the first frame image and the third frame image that are used to calculate the motion vector. In, in the second frame image (() of) and the third frame image (() of), the outline of the subjectin the first frame image (() of) is indicated by a broken line.

730 4 730 4 15 FIG. 15 FIG. The motion vectoris calculated for each pixel from the first frame image and the third frame image (see () of). The motion vectoris indicated by an arrow in () of.

730 The length of the calculated motion vectoris adjusted according to the ratio of the time difference between a time at which the first frame image is captured and a time at which the second frame image is captured to the time difference between the time at which the first frame image is captured and a time at which the third frame image is captured.

721 730 731 5 5 15 FIG. 15 FIG. A virtual turning-off state exposure frame imageis calculated by performing a correction to shift each pixel of the first frame image in the same direction as the motion vectorby the length of the motion vectorafter the length adjustment (see () of). The shift direction and shift amount of each pixel in the correction are indicated by arrows in the diagram of () of.

721 721 6 15 FIG. In order to reduce the influence of edge noise due to movement error in shift of each pixel when calculating the virtual turning-off state exposure frame image, a filter may be applied to the virtual turning-off state exposure frame image(see () of).

721 7 15 FIG. A process of subtracting the virtual turning-off state exposure frame imagefrom the second frame image is performed (see () of).

16 FIG. 1 FIG. 1 is a flowchart illustrating an example of operations of the imaging deviceof, according to an embodiment.

601 605 130 100 210 200 130 130 220 130 601 210 220 210 602 605 The operations S-Sof the flowchart may be executed, according to a program, by the cooperative operation of the controllerof the imaging unitand the controllerof the image processing device. In some embodiments, the controllermay access a first program stored in a memory of the controlleror stored in the memory, and execute the first program to cause the controllerto perform operation S. In some embodiments, the controllermay access a second program stored in the memoryand execute the second program to cause the controllerto perform operations S-S.

130 120 110 601 130 200 140 The controllercaptures the first frame image, the second frame image, and the third frame image, by the camera, in synchronization with the turning-off, turning-on, and turning-off of the light emitter(S). For example, the first frame image may refer to the first turning-off state exposure frame image. The second frame image may refer to the first turning-on state exposure frame image. The third frame image may refer to the second turning-off state exposure frame image. The controllertransmits a first frame image, a second frame image, and a third frame image to the image processing devicethrough the communicator.

210 730 602 210 230 730 The controllercalculates a motion vectorfrom the first frame image and the third frame image (S). For example, the controllerreceives the first frame image, the second frame image, and the third frame image through the communicatorand calculates a motion vectorfrom the first frame image and the third frame image.

210 730 603 The controlleradjusts the length of the motion vectoraccording to the ratio of the time difference between a time at which the first frame image is captured and a time at which the second frame image is captured to the time difference between the time at which the first frame image is captured and a time at which the third frame image is captured (S).

210 730 731 721 604 The controllershifts each pixel of the first frame image in the same direction as the motion vectorby the length of the motion vectorafter the length adjustment and calculates a virtual turning-off state exposure frame image(S).

210 721 605 The controllerperforms a process of subtracting the virtual turning-off state exposure frame imagefrom the second frame image (S).

15 16 FIGS.- 710 120 710 730 730 720 According to the embodiment illustrated in, when the background light is strong and the turning-on state exposure frame imageis close to the saturation level of the camera, and thus, it is difficult to obtain the contrast difference of the characteristic amount of the turning-on state exposure frame image, the precision of the motion vectormay be improved by calculating the motion vectorusing two turning-off state exposure frame images.

15 16 FIGS.- 15 16 FIGS.- 721 720 720 730 720 720 720 721 720 720 In the embodiment illustrated with respect to, a virtual turning-off state exposure frame imageis generated by shifting each pixel of the turning-off state exposure frame imagecaptured first from among two turning-off state exposure frame imagesused to calculate the motion vector. For example, the turning-off state exposure frame imagecaptured first may be a turning-off state exposure frame imageused for processing. In an embodiment, each pixel of the turning-off state exposure frame imagecaptured later is shifted to generate a virtual turning-off state exposure frame image. For example, the turning-off state exposure frame imagecaptured later may be a turning-off state exposure frame imageused for processing. In other respects, the embodiment is the same as the embodiment illustrated with respect to, and thus, redundant descriptions are omitted or simplified for conciseness.

17 FIG. 730 730 721 721 710 is a diagram for explaining an example of calculating a motion vector, adjusting the length of the motion vector, calculating a virtual turning-off state exposure frame image, and subtracting the virtual turning-off state exposure frame imagefrom a turning-on state exposure frame image, according to an embodiment.

1 2 3 720 710 720 700 710 720 700 730 730 2 3 500 1 17 FIG. 17 FIG. 17 FIG. 17 FIG. 17 FIG. 17 FIG. 17 FIG. The first frame image (() of), the second frame image (() of), and the third frame image (() of) are a turning-off state exposure frame image, a turning-on state exposure frame image, and a turning-off state exposure frame image, respectively. These frame imagesare captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first turning-off state exposure frame image, a first turning-on state exposure frame image, and a second turning-off state exposure frame image, respectively. The second frame image is the turning-on state exposure frame imageof a processing target. The third frame image is the turning-off state exposure frame imageused for the processing. The first frame image and the third frame image are two frame imagesused to calculate a motion vector. The second frame image, which is a processing target, is interpolated into the first frame image and the third frame image that are used to calculate the motion vector. In, in the second frame image (() of) and the third frame image (() of), the outline of the subjectin the first frame image (() of) is indicated by a broken line.

730 4 730 4 17 FIG. 17 FIG. The motion vectoris calculated for each pixel from the first frame image and the third frame image (see () of). The motion vectoris indicated by an arrow in () of.

730 The length of the calculated motion vectoris adjusted according to the ratio of the time difference between a time at which the second frame image is captured and a time at which the third frame image is captured to the time difference between a time at which the first frame image is captured and the time at which the third frame image is captured.

721 730 731 5 5 17 FIG. 17 FIG. A virtual turning-off state exposure frame imageis calculated by performing a correction to shift each pixel of the third frame image in the opposite direction to the motion vectorby the length of the motion vectorafter the length adjustment (see () of). The shift direction and shift amount of each pixel in the correction are indicated by arrows in the diagram of () of.

721 721 6 17 FIG. In order to reduce the influence of edge noise due to movement error in shift of each pixel when calculating the virtual turning-off state exposure frame image, a filter may be applied to the virtual turning-off state exposure frame image(see () of).

721 7 17 FIG. A process of subtracting the virtual turning-off state exposure frame imagefrom the second frame image is performed (see () of).

18 FIG. 1 FIG. 1 is a flowchart illustrating an example of operations of the imaging deviceof, according to an embodiment.

701 705 130 100 210 200 130 130 220 130 701 210 220 210 702 705 The operations Sto Sof the flowchart may be executed, according to a program, by the cooperative operation of the controllerof the imaging unitand the controllerof the image processing device. In some embodiments, the controllermay access a first program stored in a memory of the controlleror stored in the memory, and execute the first program to cause the controllerto perform operation S. In some embodiments, the controllermay access a second program stored in the memoryand execute the second program to cause the controllerto perform operations S-S.

130 120 110 701 130 200 140 The controllercaptures the first frame image, the second frame image, and the third frame image, by the camera, in synchronization with the turning-off, turning-on, and turning-off of the light emitter(S). For example, the first frame image may refer to the first turning-off state exposure frame image. The second frame image may refer to the first turning-on state exposure frame image. The third frame image may refer to the second turning-off state exposure frame image. The controllertransmits a first frame image, a second frame image, and a third frame image to the image processing devicethrough the communicator.

210 730 702 210 230 730 The controllercalculates a motion vectorfrom the first frame image and the third frame image (S). For example, the controllerreceives the first frame image, the second frame image, and the third frame image through the communicatorand calculates a motion vectorfrom the first frame image and the third frame image.

210 730 703 The controlleradjusts the length of the motion vectoraccording to the ratio of the time difference between a time at which the second frame image is captured and a time at which the third frame image is captured to the time difference between a time at which the first frame image is captured and the time at which the third frame image is captured (S).

210 730 731 721 704 The controllershifts each pixel of the third frame image in the opposite direction to the motion vectorby the length of the motion vectorafter the length adjustment and calculates a virtual turning-off state exposure frame image(S).

210 721 705 The controllerperforms a process of subtracting the virtual turning-off state exposure frame imagefrom the second frame image (S).

17 18 FIGS.- 17 18 FIGS.- 721 720 720 710 710 720 710 720 According to the embodiment illustrated with respect to, in the generation of the virtual turning-off state exposure frame image, each pixel of the turning-off state exposure frame imageis shifted in order to return a turning-off state exposure frame imagecaptured later to a time point of exposure of a turning-on state exposure frame imagecaptured first. The embodiment illustrated with respect tocan be suitably employed in a case where the turning-on state exposure frame imageof a processing target is captured earlier than the turning-off state exposure frame imageused for processing and the capturing times of the two imagesandare relatively similar to each other.

15 16 FIGS.- 15 16 FIGS.- 15 16 FIGS.- 17 18 FIGS.- 721 720 720 710 710 720 710 720 In the embodiment illustrated with respect to, in the generation of the virtual turning-off state exposure frame image, each pixel of the turning-off state exposure frame imageis shifted in order to advance a turning-off state exposure frame imagecaptured first to a time point of exposure of a turning-on state exposure frame imagecaptured later. The embodiment illustrated with respect tocan be suitably employed in a case where the turning-on state exposure frame imageof a processing target is captured later than the turning-off state exposure frame imageused for processing and the capturing times of the two imagesandare relatively similar to each other. Both the embodiment illustrated with respect toand the embodiment illustrated with respect tomay be flexibly employed depending on a capture situation or a capture control method.

1 6 FIGS.- 1 6 FIGS.- 710 720 710 721 720 710 730 710 710 720 721 720 710 730 In the embodiment illustrated with respect to, frame images are captured in the order of a turning-on state exposure frame image, a turning-off state exposure frame image, and a turning-on state exposure frame image, and a virtual turning-off state exposure frame imageis generated by shifting each pixel of a turning-off state exposure frame imageinterpolated into two turning-on state exposure frame imagesused to calculate the moving vector. In an embodiment, frame images are captured in the order of a turning-on state exposure frame image, a turning-on state exposure frame image, and a turning-off state exposure frame image, and a virtual turning-off state exposure frame imageis generated by shifting each pixel of a turning-off state exposure frame imageextrapolated to two turning-on state exposure frame imagesused to calculate the moving vector. In other respects, the embodiment is the same as the embodiment illustrated with respect to, and thus, redundant descriptions are omitted or simplified for conciseness.

19 FIG. 710 730 720 is a diagram for explaining an example of two turning-on state exposure frame imagesused to calculate a motion vectorand a turning-off state exposure frame imageused for processing, according to an embodiment.

130 110 130 120 710 720 An illumination control signal is a control signal for the controllerto turn the light emitteron or off. An imaging element frame signal is a control signal for the controllerto expose the cameraand read an image signal from each element (pixel). In the present embodiment, after two turning-on state exposure frame imagesare sequentially captured, two turning-off state exposure frame imagesmay be sequentially captured.

700 710 720 710 700 730 19 FIG. The first frame image, the second frame image, and the third frame image are frame imagesthat are sequentially captured in time. In, the first frame image and the second frame image are turning-on state exposure frame images. The third frame image is a turning-off state exposure frame imageThe second frame image may be a turning-on state exposure frame imagesof a processing target, the third frame image may be a turning-off state exposure frame image used for processing, and the first frame image and the second frame image may be two frame imagesused to calculate the motion vector.

20 FIG. 730 730 721 721 710 is a diagram for explaining an example of calculating a motion vector, adjusting the length of the motion vector, calculating a virtual turning-off state exposure frame image, and subtracting the virtual turning-off state exposure frame imagefrom a turning-on state exposure frame image, according to an embodiment.

1 2 3 710 710 720 700 710 720 700 730 730 2 3 500 1 20 FIG. 20 FIG. 20 FIG. 20 FIG. 20 FIG. 20 FIG. 20 FIG. The first frame image (() of), the second frame image (() of), and the third frame image (() of) are a turning-on state exposure frame image, a turning-on state exposure frame image, and a turning-off state exposure frame image, respectively. These frame imagesare captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first turning-on state exposure frame image, a second turning-on state exposure frame image, and a first turning-off state exposure frame image, respectively. The second frame image is the turning-on state exposure frame imageof a processing target. The third frame image is the turning-off state exposure frame imageused for the processing. The first frame image and the second frame image are two frame imagesused to calculate a motion vector. The third frame image used for processing is extrapolated to the first frame image and the second frame image that are used to calculate the motion vector. In, in the second frame image (() of) and the third frame image (() of), the outline of the subjectin the first frame image (() of) is indicated by a broken line.

730 4 730 4 20 FIG. 20 FIG. The motion vectoris calculated for each pixel from the first frame image and the second frame image (see () of). The motion vectoris indicated by an arrow in () of.

730 The length of the calculated motion vectoris adjusted according to the ratio of the time difference between a time at which the second frame image is captured and a time at which the third frame image is captured to the time difference between a time at which the first frame image is captured and the time at which the third frame image is captured.

721 730 731 5 5 20 FIG. 20 FIG. A virtual turning-off state exposure frame imageis calculated (or made) by performing a correction to shift each pixel of the third frame image in the opposite direction to the motion vectorby the length of the motion vectorafter the length adjustment (see () of). The shift direction and shift amount of each pixel in the correction are indicated by arrows in the diagram of () of.

721 721 6 20 FIG. In order to reduce the influence of edge noise due to movement error in shift of each pixel when calculating the virtual turning-off state exposure frame image, a filter may be applied to the virtual turning-off state exposure frame image(see () of).

721 7 20 FIG. A process of subtracting the virtual turning-off state exposure frame imagefrom the second frame image is performed (see () of).

21 FIG. 1 FIG. 1 is a flowchart illustrating an example of operations of the imaging deviceof, according to an embodiment.

801 805 130 100 210 200 130 130 220 130 801 210 220 210 802 805 The operations Sto Sof the flowchart may be executed, according to a program, by the cooperative operation of the controllerof the imaging unitand the controllerof the image processing device. In some embodiments, the controllermay access a first program stored in a memory of the controlleror stored in the memory, and execute the first program to cause the controllerto perform operation S. In some embodiments, the controllermay access a second program stored in the memoryand execute the second program to cause the controllerto perform operations S-S.

130 120 110 801 130 200 140 The controllercaptures the first frame image, the second frame image, and the third frame image, by the camera, in synchronization with the turning-on, turning-on, and turning-off of the light emitter(S). For example, the first frame image may refer to the first turning-on state exposure frame image. The second frame image may refer to the second turning-on state exposure frame image. The third frame image may refer to the first turning-off state exposure frame image. The controllertransmits a first frame image, a second frame image, and a third frame image to the image processing devicethrough the communicator.

210 730 802 210 230 730 The controllercalculates a motion vectorfrom the first frame image and the second frame image (S). For example, the controllerreceives the first frame image, the second frame image, and the third frame image through the communicatorand calculates a motion vectorfrom the first frame image and the second frame image.

210 730 803 The controlleradjusts the length of the motion vectoraccording to the ratio of the time difference between a time at which the second frame image is captured and a time at which the third frame image is captured to the time difference between a time at which the first frame image is captured and the time at which the second frame image is captured (S).

210 730 731 721 804 The controllershifts each pixel of the third frame image in the opposite direction to the motion vectorby the length of the motion vectorafter the length adjustment and calculates a virtual turning-off state exposure frame image(S).

210 721 805 The controllerperforms a process of subtracting the virtual turning-off state exposure frame imagefrom the second frame image (S).

19 21 FIGS.- 110 110 720 710 721 According to the embodiment illustrated with respect to, as two exposures in which the turning-on of the light emitteris continuous and two exposures in which the turning-off of the light emitteris continuous are alternately repeated, even when the turning-off state exposure frame imageis not captured during the capturing of two turning-on state exposure frame images, the virtual turning-off state exposure frame imagemay be appropriately calculated.

19 21 FIGS.- 19 21 FIGS.- 710 710 720 721 730 720 710 730 720 710 710 721 730 720 710 730 In the embodiment illustrated with respect to, frame images are captured in the order of a turning-on state exposure frame image, a turning-on state exposure frame image, and a turning-off state exposure frame image, and a virtual turning-off state exposure frame imageis generated by shifting, in the opposite direction to the motion vector, each pixel of a turning-off state exposure frame imageextrapolated to two turning-on state exposure frame imagesused to calculate the moving vector. In an embodiment, frame images are captured in the order of a turning-off state exposure frame image, a turning-on state exposure frame image, and a turning-on state exposure frame image, and a virtual turning-off state exposure frame imageis generated by shifting, in the same direction as the motion vector, each pixel of a turning-off state exposure frame imageextrapolated to two turning-on state exposure frame imagesused to calculate the moving vector. In other respects, the embodiment is the same as the embodiment illustrated with respect to, and thus, redundant descriptions are omitted or simplified for conciseness.

22 FIG. 730 730 721 721 710 is a diagram for explaining an example of calculating a motion vector, adjusting the length of the motion vector, calculating a virtual turning-off state exposure frame image, and subtracting the virtual turning-off state exposure frame imagefrom a turning-on state exposure frame image, according to an embodiment.

1 2 3 720 710 710 700 710 720 700 730 730 2 3 500 1 22 FIG. 22 FIG. 22 FIG. 22 FIG. 22 FIG. 22 FIG. 22 FIG. The first frame image (() of), the second frame image (() of), and the third frame image (() of) are a turning-off state exposure frame image, a turning-on state exposure frame image, and a turning-on state exposure frame image, respectively. These frame imagesare captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first turning-off state exposure frame image, a first turning-on state exposure frame image, and a second turning-on state exposure frame image, respectively. The second frame image is the turning-on state exposure frame imageof a processing target. The first frame image is the turning-off state exposure frame imageused for the processing. The second frame image and the third frame image are two frame imagesused to calculate a motion vector. The first frame image used for processing is extrapolated to the second frame image and the third frame image that are used to calculate the motion vector. In, in the second frame image (() of) and the third frame image (() of), the outline of the subjectin the first frame image (() of) is indicated by a broken line.

730 4 730 4 22 FIG. 22 FIG. The motion vectoris calculated for each pixel from the second frame image and the third frame image (see () of). The motion vectoris indicated by an arrow in () of.

730 The length of the calculated motion vectoris adjusted according to the ratio of the time difference between a time at which the first frame image is captured and a time at which the second frame image is captured to the time difference between the time at which the second frame image is captured and a time at which the third frame image is captured.

721 730 731 5 5 22 FIG. 22 FIG. A virtual turning-off state exposure frame imageis calculated (or made) by performing a correction to shift each pixel of the first frame image in the same direction as the motion vectorby the length of the motion vectorafter the length adjustment (see () of). The shift direction and shift amount of each pixel in the correction are indicated by arrows in the diagram of () of.

721 721 6 22 FIG. In order to reduce the influence of edge noise due to movement error in shift of each pixel when calculating the virtual turning-off state exposure frame image, a filter may be applied to the virtual turning-off state exposure frame image(see () of).

721 7 22 FIG. A process of subtracting the virtual turning-off state exposure frame imagefrom the second frame image is performed (see () of).

23 FIG. 1 FIG. 1 is a flowchart illustrating an example of operations of the imaging deviceof, according to an embodiment.

901 905 130 100 210 200 130 130 220 130 901 210 220 210 902 905 The operations Sto Sof the flowchart may be executed, according to a program, by the cooperative operation of the controllerof the imaging unitand the controllerof the image processing device. In some embodiments, the controllermay access a first program stored in a memory of the controlleror stored in the memory, and execute the first program to cause the controllerto perform operation S. In some embodiments, the controllermay access a second program stored in the memoryand execute the second program to cause the controllerto perform operations S-S.

130 120 110 901 130 200 140 The controllercaptures the first frame image, the second frame image, and the third frame image, by the camera, in synchronization with the turning-off, turning-on, and turning-on of the light emitter(S). For example, the first frame image may refer to the first turning-off state exposure frame image. The second frame image may refer to the first turning-on state exposure frame image. The third frame image may refer to the second turning-on state exposure frame image. The controllertransmits a first frame image, a second frame image, and a third frame image to the image processing devicethrough the communicator.

210 730 902 210 230 730 The controllercalculates a motion vectorfrom the second frame image and the third frame image (S). For example, the controllerreceives the first frame image, the second frame image, and the third frame image through the communicatorand calculates a motion vectorfrom the second frame image and the third frame image.

210 730 903 The controlleradjusts the length of the motion vectoraccording to the ratio of the time difference between a time at which the first frame image is captured and a time at which the second frame image is captured to the time difference between the time at which the second frame image is captured and a time at which the third frame image is captured (S).

210 730 731 721 904 The controllershifts each pixel of the first frame image in the same direction as the motion vectorby the length of the motion vectorafter the length adjustment and calculates a virtual turning-off state exposure frame image(S).

210 721 905 The controllerperforms a process of subtracting the virtual turning-off state exposure frame imagefrom the second frame image (S).

22 23 FIGS.- 19 21 FIGS.- 22 23 FIGS.- 19 21 FIGS.- 110 110 720 710 721 500 According to the embodiment illustrated with respect to, as in the embodiment illustrated with respect to, as two exposures in which the turning-on of the light emitteris continuous and two exposures in which the turning-off of the light emitteris continuous are alternately repeated, even when the turning-off state exposure frame imageis not captured during the capturing of two turning-on state exposure frame images, the virtual turning-off state exposure frame imagemay be appropriately calculated. In addition, the processing method may be optimized by selecting the embodiment illustrated with respect toor the embodiment illustrated with respect tobased on the movement trajectory of the subject.

19 21 FIGS.- 19 21 FIGS.- 710 710 720 721 730 720 710 730 720 720 710 721 730 720 710 720 730 In the embodiment illustrated with respect to, frame images are captured in the order of a turning-on state exposure frame image, a turning-on state exposure frame image, and a turning-off state exposure frame image, and a virtual turning-off state exposure frame imageis generated by shifting, in the opposite direction to the motion vector, each pixel of a turning-off state exposure frame imageextrapolated to two turning-on state exposure frame imagesused to calculate the moving vector. In an embodiment, frame images are captured in the order of a turning-off state exposure frame image, a turning-off state exposure frame image, and a turning-on state exposure frame image, and a virtual turning-off state exposure frame imageis generated by shifting, in the same direction as the motion vector, each pixel of a turning-off state exposure frame imageused to process a turning-on state exposure frame imageextrapolated to two turning-off state exposure frame imagesused to calculate the motion vector. In other respects, the embodiment is the same as the embodiment illustrated with respect to, and thus, redundant descriptions are omitted or simplified for conciseness.

24 FIG. 730 730 721 721 710 is a diagram for explaining an example of calculating a motion vector, adjusting the length of the motion vector, calculating a virtual turning-off state exposure frame image, and subtracting the virtual turning-off state exposure frame imagefrom a turning-on state exposure frame image, according to an embodiment.

1 2 3 720 720 710 700 710 720 700 730 730 2 3 500 1 24 FIG. 24 FIG. 24 FIG. 24 FIG. 24 FIG. 24 FIG. 24 FIG. The first frame image (() of), the second frame image (() of), and the third frame image (() of) are a turning-off state exposure frame image, a turning-off state exposure frame image, and a turning-on state exposure frame image, respectively. These frame imagesare captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first turning-off state exposure frame image, a second turning-off state exposure frame image, and a first turning-on state exposure frame image, respectively. The third frame image is the turning-on state exposure frame imageof a processing target. The second frame image is the turning-off state exposure frame imageused for the processing. The first frame image and the second frame image are two frame imagesused to calculate a motion vector. The third frame image that is a processing target is extrapolated to the first frame image and the second frame image that are used to calculate the motion vector. In, in the second frame image (() of) and the third frame image (() of), the outline of the subjectin the first frame image (() of) is indicated by a broken line.

730 4 730 4 24 FIG. 24 FIG. The motion vectoris calculated for each pixel from the first frame image and the second frame image (see () of). The motion vectoris indicated by an arrow in () of.

730 The length of the calculated motion vectoris adjusted according to the ratio of the time difference between a time at which the second frame image is captured and a time at which the third frame image is captured to the time difference between a time at which the first frame image is captured and the time at which the second frame image is captured.

721 730 731 5 5 24 FIG. 24 FIG. A virtual turning-off state exposure frame imageis calculated by performing a correction to shift each pixel of the second frame image in the same direction as the motion vectorby the length of the motion vectorafter the length adjustment (see () of). The shift direction and shift amount of each pixel in the correction are indicated by arrows in the diagram of () of.

721 721 6 24 FIG. In order to reduce the influence of edge noise due to movement error in shift of each pixel when calculating the virtual turning-off state exposure frame image, a filter may be applied to the virtual turning-off state exposure frame image(see () of).

721 7 24 FIG. A process of subtracting the virtual turning-off state exposure frame imagefrom the third frame image is performed (see () of).

25 FIG. 1 FIG. 1 is a flowchart illustrating an example of operations of the imaging deviceof, according to an embodiment.

1001 1005 130 100 210 200 130 130 220 130 1001 210 220 210 1002 1005 The operations Sto Sof the flowchart may be executed, according to a program, by the cooperative operation of the controllerof the imaging unitand the controllerof the image processing device. In some embodiments, the controllermay access a first program stored in a memory of the controlleror stored in the memory, and execute the first program to cause the controllerto perform operation S. In some embodiments, the controllermay access a second program stored in the memoryand execute the second program to cause the controllerto perform operations S-S.

130 120 110 1001 130 200 140 The controllercaptures the first frame image, the second frame image, and the third frame image, by the camera, in synchronization with the turning-off, turning-off, and turning-on of the light emitter(S). For example, the first frame image may refer to the first turning-off state exposure frame image. The second frame image may refer to the second turning-off state exposure frame image. The third frame image may refer to the first turning-on state exposure frame image. The controllertransmits a first frame image, a second frame image, and a third frame image to the image processing devicethrough the communicator.

210 730 1002 210 230 730 The controllercalculates a motion vectorfrom the first frame image and the second frame image (S). For example, the controllerreceives the first frame image, the second frame image, and the third frame image through the communicatorand calculates a motion vectorfrom the first frame image and the second frame image.

210 730 1003 The controlleradjusts the length of the motion vectoraccording to the ratio of the time difference between a time at which the second frame image is captured and a time at which the third frame image is captured to the time difference between a time at which the first frame image is captured and the time at which the second frame image is captured (S).

210 730 731 721 1004 The controllershifts each pixel of the second frame image in the same direction as the motion vectorby the length of the motion vectorafter the length adjustment and calculates a virtual turning-off state exposure frame image(S).

210 721 1005 The controllerperforms a process of subtracting the virtual turning-off state exposure frame imagefrom the third frame image (S).

24 25 FIGS.- 110 110 710 120 710 730 730 720 According to the embodiment illustrated with respect to, even when a plurality of exposures, in which the turning-on of the light emitteris continuous, and a plurality of exposures, in which the turning-off of the light-emitteris continuous, are alternately repeated, the background light is strong, and the turning-on state exposure frame imageis close to the saturation level of the camera, making it difficult to obtain a contrast difference in the characteristic amount of the turning-on state exposure frame image, the precision of the motion vectormay be improved by calculating the motion vectorusing two turning-off state exposure frame images.

24 25 FIGS.- 24 25 FIGS.- 720 720 710 721 730 720 710 720 730 710 720 720 721 730 720 710 720 730 In the embodiment illustrated with respect to, frame images are captured in the order of a turning-off state exposure frame image, a turning-off state exposure frame image, and a turning-on state exposure frame image, and a virtual turning-off state exposure frame imageis generated by shifting, in the same direction as the motion vector, each pixel of a turning-off state exposure frame imageused to process a turning-on state exposure frame imageextrapolated to two turning-off state exposure frame imagesused to calculate the motion vector. In an embodiment, frame images are captured in the order of a turning-on state exposure frame image, a turning-off state exposure frame image, and a turning-off state exposure frame image, and a virtual turning-off state exposure frame imageis generated by shifting, in the opposite direction to the motion vector, each pixel of a turning-off state exposure frame imageused to process a turning-on state exposure frame imageextrapolated to two turning-off state exposure frame imagesused to calculate the motion vector. In other respects, the embodiment is the same as the embodiment illustrated with respect to, and thus, redundant descriptions are omitted or simplified for conciseness.

26 FIG. 730 730 721 721 710 is a diagram for explaining an example of calculating a motion vector, adjusting the length of the motion vector, calculating a virtual turning-off state exposure frame image, and subtracting the virtual turning-off state exposure frame imagefrom a turning-on state exposure frame image, according to an embodiment.

1 2 3 710 720 720 700 710 720 700 730 730 2 3 500 1 26 FIG. 26 FIG. 26 FIG. 26 FIG. 26 FIG. 26 FIG. 26 FIG. The first frame image (() of), the second frame image (() of), and the third frame image (() of) are a turning-on state exposure frame image, a turning-off state exposure frame image, and a turning-off state exposure frame image, respectively. These frame imagesare captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first turning-on state exposure frame image, a first turning-off state exposure frame image, and a second turning-off state exposure frame image, respectively. The first frame image is the turning-on state exposure frame imageof a processing target. The second frame image is the turning-off state exposure frame imageused for the processing. The second frame image and the third frame image are two frame imagesused to calculate a motion vector. The first frame image that is a processing target is extrapolated to the second frame image and the third frame image that are used to calculate the motion vector. In, in the second frame image (() of) and the third frame image (() of), the outline of the subjectin the first frame image (() of) is indicated by a broken line.

730 4 730 4 26 FIG. 26 FIG. The motion vectoris calculated for each pixel from the second frame image and the third frame image (see () of). The motion vectoris indicated by an arrow in () of.

730 The length of the calculated motion vectoris adjusted according to the ratio of the time difference between a time at which the first frame image is captured and a time at which the second frame image is captured to the time difference between the time at which the second frame image is captured and a time at which the third frame image is captured.

721 730 731 5 5 26 FIG. 26 FIG. A virtual turning-off state exposure frame imageis calculated (or made) by performing a correction to shift each pixel of the second frame image in the opposite direction to the motion vectorby the length of the motion vectorafter the length adjustment (see () of). The shift direction and shift amount of each pixel in the correction are indicated by arrows in the diagram of () of.

721 721 6 26 FIG. In order to reduce the influence of edge noise due to movement error in shift of each pixel when calculating the virtual turning-off state exposure frame image, a filter may be applied to the virtual turning-off state exposure frame image(see () of).

721 7 26 FIG. A process of subtracting the virtual turning-off state exposure frame imagefrom the third frame image is performed (see () of).

27 FIG. 1 FIG. 1 is a flowchart illustrating an example of operations of the imaging deviceof, according to an embodiment.

1101 1105 130 100 210 200 130 130 220 130 1101 210 220 210 1102 1105 The operations Sto Sof the flowchart may be executed, according to a program, by the cooperative operation of the controllerof the imaging unitand the controllerof the image processing device. In some embodiments, the controllermay access a first program stored in a memory of the controlleror stored in the memory, and execute the first program to cause the controllerto perform operation S. In some embodiments, the controllermay access a second program stored in the memoryand execute the second program to cause the controllerto perform operations S-S.

130 120 110 1101 130 200 140 The controllercaptures the first frame image, the second frame image, and the third frame image, by the camera, in synchronization with the turning-on, turning-off, and turning-off of the light emitter(S). For example, the first frame image may refer to the first turning-on state exposure frame image. The second frame image may refer to the first turning-off state exposure frame image. The third frame image may refer to the second turning-off state exposure frame image. The controllertransmits a first frame image, a second frame image, and a third frame image to the image processing devicethrough the communicator.

210 730 1102 210 230 730 The controllercalculates a motion vectorfrom the second frame image and the third frame image (S). For example, the controllerreceives the first frame image, the second frame image, and the third frame image through the communicatorand calculates a motion vectorfrom the second frame image and the third frame image.

210 730 1103 The controlleradjusts the length of the motion vectoraccording to the ratio of the time difference between a time at which the first frame image is captured and a time at which the second frame image is captured to the time difference between the time at which the second frame image is captured and a time at which the third frame image is captured (S).

210 730 731 721 1104 The controllershifts each pixel of the second frame image in the opposite direction to the motion vectorby the length of the motion vectorafter the length adjustment and calculates a virtual turning-off state exposure frame image(S).

210 721 1105 The controllerperforms a process of subtracting the virtual turning-off state exposure frame imagefrom the first frame image (S).

26 27 FIGS.- 24 25 FIGS.- 24 25 FIGS.- 26 27 FIGS.- 110 110 710 120 710 730 730 720 500 According to the embodiment illustrated with respect to, as in the embodiment illustrated with respect to, even when a plurality of exposures, in which the turning-on of the light emitteris continuous, and a plurality of exposures, in which the turning-off of the light-emitteris continuous, are alternately repeated, the background light is strong, and the turning-on state exposure frame imageis close to the saturation level of the camera, making it difficult to obtain a contrast difference in the characteristic amount of the turning-on state exposure frame image, the precision of the motion vectormay be improved by calculating the motion vectorusing two turning-off state exposure frame images. In addition, the processing method can be optimized by selecting the embodiment illustrated with respect toor the embodiment illustrated with respect tobased on the movement trajectory of the subject ().

1 6 FIGS.- 1 6 FIGS.- 700 700 In the embodiment illustrated with respect to, each frame image (each of the first to third frame images) is captured by one exposure. In an embodiment, each frame image(each of the first to third frame images) is captured by adding a plurality of frame images captured by a plurality of exposures to generate one frame image. In other respects, the embodiment is the same as the embodiment illustrated with respect to, and thus, redundant descriptions are omitted or simplified for conciseness.

28 FIG. 710 720 is a diagram for explaining the capturing of a plurality of frame images by a plurality of exposures and the capturing of a turning-on state exposure frame imageand a turning-off state exposure frame imageby the addition of the plurality of frame images, according to an embodiment.

120 710 700 110 110 120 720 700 110 110 120 700 110 120 700 110 120 700 110 The cameracaptures a turning-on state exposure frame imageby adding digital signals of a plurality of frame imagescaptured by continuously multiple times performing the turning-on of the light emitterand exposure synchronized with the turning-on of the light emitter. The cameracaptures a turning-off state exposure frame imageby adding digital signals of a plurality of frame imagescaptured by continuously multiple times performing the turning-off of the light emitterand exposure synchronized with the turning-off of light emitter. That is, the cameracaptures a first frame image as a post-addition frame imageby adding a plurality of frame images captured by exposure synchronized with the turning-on of the light emitter. Next, the cameracaptures a second frame image as a post-addition frame imageby adding a plurality of frame images captured by exposure synchronized with the turning-off of the light emitter. Next, the cameracaptures a third frame image as a post-addition frame imageby adding a plurality of frame images captured by exposure synchronized with the turning-on of the light emitter.

28 FIG. In the embodiment illustrated with respect to, the time of the exposure time is set to a relatively short time that does not cause pixel saturation of an imaging element. The time of the exposure may be appropriately set by experiment from the viewpoint of avoiding the pixel saturation of the imaging element.

700 120 130 100 210 200 The addition of the plurality of frame imagesdescribed above is performed outside the image element. The addition may be performed by the camerahaving a CPU or the like, may be performed by the controllerof the imaging unit, or may be performed by the controllerof the image processing device.

28 FIG. 710 720 According to the embodiment illustrated with respect to, even when the background light is relatively strong, a turning-on state exposure frame imageand a turning-off state exposure frame imagethat are not pixel-saturated may be captured.

28 FIG. 28 FIG. 700 700 700 In the embodiment illustrated with respect to, each frame image(each of the first to third frame images) is captured by adding a plurality of frame images captured by a plurality of exposures to generate one frame image. In an embodiment, each frame image(each of the first to third frame images) is captured by continuously multiple times performing the accumulation of a signal amount for each pixel by exposure and the integration of an accumulated signal amount to generate one frame image. In other respects, the embodiment is the same as the embodiment illustrated with respect to, and thus, redundant descriptions are omitted or simplified for conciseness.

29 FIG. 710 720 is a diagram for explaining the capturing of a turning-on state exposure frame imageby the integration of a signal amount by exposure when a plurality of turning-on operations are performed, and the capturing of a turning-off state exposure frame imageby the integration of a signal amount by exposure when a plurality of turning-off operations are performed, according to an embodiment.

120 710 110 110 120 720 110 110 120 110 120 110 120 110 The cameracaptures a turning-on state exposure frame imageby continuously multiple times performing the turning-on of the light emitter, the accumulation of a signal amount for each pixel by exposure synchronized with the turning-on of the light emitter, and the integration of an accumulated signal amount. The cameracaptures a turning-off state exposure frame imageby continuously multiple times performing the turning-off of the light emitter, the accumulation of a signal amount for each pixel by exposure synchronized with the turning-off of the light emitter, and the integration of an accumulated signal amount. That is, the cameracaptures a first frame image by integrating a signal amount accumulated by a plurality of exposures synchronized with the turning-on of the light emitter. Next, the cameracaptures a second frame image by integrating a signal amount accumulated by a plurality of exposures synchronized with the turning-off of the light emitter. Next, the cameracaptures a third frame image by integrating a signal amount accumulated by a plurality of exposures synchronized with the turning-on of the light emitter.

The above-described integration of the signal amount may be performed by accumulating the charge of the signal amount generated by the plurality of exposures in each of a plurality of capacitors and analogically adding charge amounts accumulated in each capacitor. The addition of the signal amount may be performed in the imaging element.

29 FIG. 710 720 According to the embodiment illustrated with respect to, even when the background light is relatively strong, a turning-on state exposure frame imageand a turning-off state exposure frame imagethat are not pixel-saturated may be captured.

30 FIG. is a diagram illustrating an example of a case where a first frame image to a third frame image are not continuous, according to an embodiment.

1 6 FIGS.- In the embodiment illustrated with respect to, the first frame image to the third frame image are continuous. In an embodiment, the first frame image to the third frame image may not be continuous.

30 FIG. 30 FIG. 30 FIG. 30 FIG. 30 FIG. 30 FIG. 1 2 5 720 2 1 710 In the embodiment illustrated in, a frame () ofand a frame () of, which are continuous, are respectively referred to as the first frame image and the second frame image. A frame () ofis referred to as the third frame image. In this way, in the case where the first frame image to the third frame image are not continuous, a turning-off state exposure frame image, which is used for processing, may be referred to as the frame () ofclose to the frame () ofthat is a turning-on state exposure frame imageof a processing target.

30 FIG. 1 12 FIGS.- The embodiment illustrated with respect tomay be applied to the embodiments illustrated with respect to.

30 FIG. 13 14 FIGS.- 30 FIG. 30 FIG. 30 FIG. 15 18 FIGS.- 4 5 720 In the case where the embodiment illustrated with respect tois applied to the embodiment illustrated with respect to, a frame () ofclose to the frame () of, which is a processing target, may be referred to as the second frame image that is a turning-off state exposure frame imageused for processing. The embodiment illustrated with respect tomay be equally applied to the embodiments illustrated with respect to.

31 FIG. is a diagram illustrating an example of a case where a first frame image to a third frame image are not continuous, according to an embodiment.

31 FIG. 2 3 5 720 3 2 710 In an embodiment illustrated with respect to, when exposure during a plurality of turning-on operations that are continuous, and exposure during a plurality of turning-off operations that are continuous are alternately performed, non-continuous frame images are selected as the first frame image to the third frame image. Specifically, a frame () and a frame () that are continuous are referred to as the first frame image and the second frame image, respectively. A frame () is referred to as the third frame image. In this way, in the case where the first frame image to the third frame image are not continuous, a turning-off state exposure frame image, which is used for processing, may be referred to as the frame () close to the frame () that is a turning-on state exposure frame imageof a processing target.

31 FIG. 1 12 FIGS.- The embodiment illustrated with respect tomay be applied to the embodiments illustrated with respect to.

31 FIG. 13 14 FIGS.- 31 FIG. 15 18 FIGS.- 4 5 720 In the case where the embodiment illustrated with respect tois applied to the embodiment illustrated with respect to, a frame () close to the frame (), which is a processing target, may be referred to as the second frame image that is a turning-off state exposure frame imageused for processing. The embodiment illustrated with respect tomay be equally applied to the embodiments illustrated with respect to.

32 FIG. is a diagram illustrating an example of a case where a first frame image to a third frame image are not continuous, according to an embodiment.

32 FIG. 1 3 4 720 2 4 3 710 720 2 4 720 In an embodiment illustrated with respect to, when exposure during a turning-on operation and exposure during a turning-off operation are alternately performed, non-continuous frame images are selected as the first frame image to the third frame image. Specifically, frame () is set as the first frame image. A frame () and a frame () that are continuous are referred to as the second frame image and the third frame image, respectively. In this way, in the case where the first frame image to the third frame image are not continuous, a turning-off state exposure frame image, which is used for processing, may be referred to as the frame () or the frame () close to the frame () that is a turning-on state exposure frame imageof a processing target. In the case where the turning-off state exposure frame imageof the frame () may not be used due to any error, the frame () may be selected as the turning-off state exposure frame image, which is used for processing.

32 FIG. 19 21 FIGS.- 32 FIG. 22 27 FIGS.- The embodiment illustrated with respect tomay be applied to the embodiment illustrated with respect to. The embodiment illustrated with respect tomay also be equally applied to the embodiments illustrated with respect to.

33 FIG. is a diagram illustrating an example of a case where a first frame image to a third frame image are continuous, according to an embodiment.

33 FIG. 1 3 700 In the embodiment illustrated in, when exposure during a plurality of continuous turning-on operations and exposure during a plurality of continuous turning-off operations are alternately performed, continuous frame images are selected as the first frame image to the third frame image. Specifically, a frame () to a frame () that are continuous are referred to as the first frame image to the third frame image, respectively. In this way, even when exposure during a plurality of continuous turning-on operations and exposure during a plurality of continuous turning-off operations are alternately performed, external light correction processing may be performed in a short time by selecting continuous frame imagesas the first frame image to the third frame image.

33 FIG. 19 27 FIGS.- The embodiment illustrated with respect tomay be applied to the embodiments illustrated with respect to.

A motion vector of each pixel is calculated from two turning-on state exposure frame images or two turning-off state exposure frame images, and the length of the motion vector is adjusted according to the time difference between the turning-on state exposure frame image of a processing target and the turning-off state exposure frame image used for processing. Then, the turning-off state exposure frame image used for the processing is corrected based on the motion vector after the adjustment to calculate a virtual turning-off state exposure frame image, and a process of subtracting the virtual turning-off state exposure frame image from the turning-on state exposure frame image of the processing target is performed. Thus, a high-resolution captured image without artifacts, in which the influence of background light is removed even when the subject moves in any direction, may be obtained.

In addition, when frame images captured in the order of a turning-on state exposure frame image, a turning-off state exposure frame image, and a turning-on state exposure frame image are respectively referred to as a first turning-on state exposure frame image, a first turning-off state exposure frame image, and a second turning-on state exposure frame image, the motion vector is calculated from the first turning-on state exposure frame image and the second turning-on state exposure frame image. The length of the motion vector of each pixel is adjusted according to the ratio of the time difference between the first turning-on state exposure frame image and the first turning-off state exposure frame image to the time difference between the first turning-on state exposure frame image and the second turning-on state exposure frame image. A virtual turning-off state exposure frame image is calculated by performing a correction to shift each pixel of the first turning-off state exposure frame image in the opposite direction to the motion vector by the length of the motion vector after the length adjustment. Then, the process of subtracting the calculated virtual turning-off state exposure frame image from the first turning-on state exposure frame image is performed. Thus, because the turning-off state exposure frame image subtracted from the turning-on state exposure frame image is shifted by pixel unit, an image without artifacts, in which the influence of background light is removed even when the subject moves in any direction and even when a camera moves, may be obtained.

In addition, when frame images captured in the order of a turning-on state exposure frame image, a turning-off state exposure frame image, and a turning-on state exposure frame image are respectively referred to as a first turning-on state exposure frame image, a first turning-off state exposure frame image, and a second turning-on state exposure frame image, the motion vector of each pixel is calculated from the first turning-on state exposure frame image and the second turning-on state exposure frame image. The length of the motion vector of each pixel is adjusted according to the ratio of the time difference between the first turning-off state exposure frame image and the second turning-on state exposure frame image to the time difference between the first turning-on state exposure frame image and the second turning-on state exposure frame image. A virtual turning-off state exposure frame image is calculated by performing a correction to shift each pixel of the first turning-off state exposure frame image by the length of the motion vector after the length adjustment in the same direction as the motion vector. Then, the process of subtracting the calculated virtual turning-off state exposure frame image from the second turning-on state exposure frame image is performed. Therefore, because the turning-off state exposure frame image subtracted from the turning-on state exposure frame image is shifted by pixel unit, an image without artifacts, in which the influence of background light is removed even when the subject moves in any direction and even when a camera moves, may be obtained.

In addition, when frame images captured in the order of a turning-off state exposure frame image, a turning-on state exposure frame image, and a turning-off state exposure frame image are respectively referred to as a first turning-off state exposure frame image, a first turning-on state exposure frame image, and a second turning-off state exposure frame image, a motion vector is calculated from the first turning-off state exposure frame image and the second turning-off state exposure frame image. The length of the motion vector of each pixel is adjusted according to the ratio of the time difference between the first turning-off state exposure frame image and the first turning-on state exposure frame image to the time difference between the first turning-off state exposure frame image and the second turning-off state exposure frame image. A virtual turning-off state exposure frame image is calculated by performing a correction to shift each pixel of the first turning-off state exposure frame image by the length of the motion vector after the length adjustment in the same direction as the motion vector. Then, a turning-on state exposure frame image processor performs the process of subtracting the calculated virtual turning-off state exposure frame image from the first turning-on state exposure frame image. Thus, even when the background light is strong and the turning-on state exposure frame image is close to the saturation level of the camera, making it difficult to obtain a contrast difference in the characteristic amount of the turning-on state exposure frame image, the precision of the motion vector may be improved by calculating the motion vector using two turning-off state exposure frame images. In addition, in a case where the turning-on state exposure frame image of a processing target is captured later than the turning-off state exposure frame image used for processing and the capturing times of the two images are relatively similar to each other, an image without artifacts, in which the influence of background light is removed more simply and effectively, may be obtained.

In addition, when frame images captured in the order of a turning-off state exposure frame image, a turning-on state exposure frame image, and a turning-off state exposure frame image are respectively referred to as a first turning-off state exposure frame image, a first turning-on state exposure frame image, and a second turning-off state exposure frame image, a motion vector is calculated from the first turning-off state exposure frame image and the second turning-off state exposure frame image. The length of the motion vector of each pixel is adjusted according to the ratio of the time difference between the first turning-on state exposure frame image and the second turning-off state exposure frame image to the time difference between the first turning-off state exposure frame image and the second turning-off state exposure frame image. A virtual turning-off state exposure frame image is calculated by performing a correction to shift each pixel of the second turning-off state exposure frame image in the opposite direction to the motion vector by the length of the motion vector after the length adjustment. Then, the process of subtracting the calculated virtual turning-off state exposure frame image from the first turning-on state exposure frame image is performed. Thus, in a case where the turning-on state exposure frame image of a processing target is captured earlier than the turning-off state exposure frame image used for processing and the capturing times of the two images are relatively similar to each other, an image without artifacts, in which the influence of background light is removed more simply and effectively, may be obtained.

In addition, a pixel of which the length of the motion vector is less than or equal to a certain threshold value is set as a certain pixel of which the length of the motion vector is not adjusted. An average of the motion vector of each pixel is calculated, and the length of the average of the motion vector is adjusted according to the time difference between the turning-on state exposure frame image of a processing target and the turning-off exposure frame image used for processing. Then, the motion vector of each pixel other than the certain pixel is adjusted by the average after the adjustment. Thus, by uniformly shifting all pixels other than the certain pixel based on the average of the length of the motion vector, the amount of calculation for processing may be effectively reduced.

In addition, a face region is detected from the turning-on state exposure frame image or two turning-off state exposure frame images, which are used to calculate the motion vector. The motion vector of each pixel within the detected face region is calculated. Thus, because a target pixel for which the motion vector is calculated, and a pixel that is a shift target in the calculation of the virtual turning-off state exposure frame image are limited to the face region, the amount of calculation may be suppressed.

In addition, the calculated virtual turning-off state exposure frame image is filtered using multiple types of low-pass filters, thereby calculating a plurality of virtual turning-off state exposure frame images after the filtering. The processing is performed to subtract, from the turning-on state exposure frame image of the processing target, the virtual turning-off state exposure frame image after the filtering, which has the smallest standard deviation of the difference between the calculated virtual turning-off state exposure frame image after the filtering and the virtual turning-off state exposure frame image before the filtering. Thus, by performing the processing using an optimally filtered virtual turning-off state exposure frame image, an edge noise due to the shift amount error of a pixel in the calculation of the virtual turning-off state exposure frame image may be further reduced.

In addition, when frame images captured in the order of a turning-on state exposure frame image, a turning-on state exposure frame image, and a turning-off state exposure frame image are respectively referred to as a first turning-on state exposure frame image, a second turning-on state exposure frame image, and a first turning-off state exposure frame image, a motion vector is calculated from the first turning-on state exposure frame image and the second turning-on state exposure frame image. The length of the motion vector of each pixel is adjusted according to the ratio of the time difference between the second turning-off state exposure frame image and the first turning-off state exposure frame image to the time difference between the first turning-on state exposure frame image and the second turning-on state exposure frame image. A virtual turning-off state exposure frame image is calculated by performing a correction to shift each pixel of the first turning-off state exposure frame image in the opposite direction to the motion vector by the length of the motion vector after the length adjustment. Then, the process of subtracting the calculated virtual turning-off state exposure frame image from the second turning-on state exposure frame image is performed. Thus, as two exposures in which the turning-on of a light emitter is continuous and two exposures in which the turning-off of the light emitter is continuous are alternately repeated, even when the turning-off state exposure frame image is not captured during the capturing of two turning-on state exposure frame images, the virtual turning-off state exposure frame image may be appropriately calculated.

In addition, when frame images captured in the order of a turning-off state exposure frame image, a turning-on state exposure frame image, and a turning-on state exposure frame image are respectively referred to as a first turning-off state exposure frame image, a first turning-on state exposure frame image, and a second turning-on state exposure frame image, a motion vector is calculated from the first turning-on state exposure frame image and the second turning-on state exposure frame image. The length of the motion vector of each pixel is adjusted according to the ratio of the time difference between the first turning-off state exposure frame image and the first turning-on state exposure frame image to the time difference between the first turning-on state exposure frame image and the second turning-on state exposure frame image. A virtual turning-off state exposure frame image is calculated by performing a correction to shift each pixel of the first turning-off state exposure frame image in the same direction as the motion vector by the length of the motion vector after the length adjustment. Then, the process of subtracting the calculated virtual turning-off state exposure frame image from the first turning-on state exposure frame image is performed. Thus, as two exposures in which the turning-on of a light emitter is continuous and two exposures in which the turning-off of the light emitter is continuous are alternately repeated, even when the turning-off state exposure frame image is not captured during the capturing of two turning-on state exposure frame images, the virtual turning-off state exposure frame image may be appropriately calculated.

In addition, when frame images captured in the order of a turning-off state exposure frame image, a turning-off state exposure frame image, and a turning-on state exposure frame image are respectively referred to as a first turning-off state exposure frame image, a second turning-off state exposure frame image, and a first turning-on state exposure frame image, a motion vector is calculated from the first turning-off state exposure frame image and the second turning-off state exposure frame image. The length of the motion vector of each pixel is adjusted according to the ratio of the time difference between the second turning-off state exposure frame image and the first turning-on state exposure frame image to the time difference between the first turning-off state exposure frame image and the second turning-off state exposure frame image. A virtual turning-off state exposure frame image is calculated by performing a correction to shift each pixel of the second turning-off state exposure frame image by the length of the motion vector after the length adjustment in the same direction as the motion vector. Then, the process of subtracting the calculated virtual turning-off state exposure frame image from the first turning-on state exposure frame image is performed. Thus, even when a plurality of exposures, in which the turning-on of a light emitter is continuous, and a plurality of exposures, in which the turning-off of the light-emitter is continuous, are alternately repeated, the background light is strong, and the turning-on state exposure frame image is close to the saturation level of a camera, making it difficult to obtain a contrast difference in the characteristic amount of the turning-on state exposure frame image, the precision of the motion vector may be improved by calculating the motion vector using two turning-off state exposure frame images.

In addition, when frame images captured in the order of a turning-on state exposure frame image, a turning-off state exposure frame image, and a turning-off state exposure frame image are respectively referred to as a first turning-on state exposure frame image, a first turning-off state exposure frame image, and a second turning-off state exposure frame image, a motion vector is calculated from the first turning-off state exposure frame image and the second turning-off state exposure frame image. The length of the motion vector of each pixel is adjusted according to the ratio of the time difference between the first turning-on state exposure frame image and the first turning-off state exposure frame image to the time difference between the first turning-off state exposure frame image and the second turning-off state exposure frame image. A virtual turning-off state exposure frame image is calculated by performing a correction to shift each pixel of the first turning-off state exposure frame image by the length of the motion vector after the length adjustment in the opposite direction to the motion vector. Then, the process of subtracting the calculated virtual turning-off state exposure frame image from the first turning-on state exposure frame image is performed. Thus, even when a plurality of exposures, in which the turning-on of a light emitter is continuous, and a plurality of exposures, in which the turning-off of the light-emitter is continuous, are alternately repeated, the background light is strong, and the turning-on state exposure frame image is close to the saturation level of a camera, making it difficult to obtain a contrast difference in the characteristic amount of the turning-on state exposure frame image, the precision of the motion vector may be improved by calculating the motion vector using two turning-off state exposure frame images.

In addition, a turning-on state exposure frame image is captured by adding digital signals of a plurality of frame images captured by continuously multiple times performing the turning-on of the light emitter and exposure synchronized with the turning-on of the light emitter. In addition, a turning-off state exposure frame image is captured by adding digital signals of a plurality of frame images captured by continuously multiple times performing the turning-off of the light emitter and exposure synchronized with the turning-off of the light emitter. Thus, even when the background light is relatively strong, a turning-on state exposure frame image and a turning-off state exposure frame image that are not pixel-saturated may be captured.

In addition, a turning-on state exposure frame image is captured by continuously multiple times performing the turning-on of the light emitter, the accumulation of a signal amount for each pixel by exposure synchronized with the turning-on of the light emitter, and the integration of an accumulated signal amount. In addition, a turning-off state exposure frame image is captured by continuously multiple times performing the turning-off of the light emitter, the accumulation of a signal amount for each pixel by exposure synchronized with the turning-off of the light emitter, and the integration of an accumulated signal amount. Thus, even when the background light is relatively strong, a turning-on state exposure frame image and a turning-off state exposure frame image that are not pixel-saturated may be captured.

While various embodiments have been particularly shown and described with reference to the drawings, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the following claims.