Light Detection Device, Light Detection Method, and Program

The present disclosure relates to a light detection device, a light detection method, and a program, and particularly relates to a light detection device, a light detection method, and a program that can reduce latency caused by an event signal detection sensor.

An event signal detection sensor has been proposed that detects an event from a comparison result between a change in an amount of light received by a pixel per unit time and a predetermined threshold, and outputs the event as event data (refer to PTL 1).

PTL 1: JP 2020-136958 A

It is known that in an event signal detection sensor, so-called latency occurs in which an occurrence time of an event that is recorded in extracted event data is delayed relative to an actual event occurrence time due to “transfer rate restrictions” or “circuit response delays that depend on an amount of light received by a pixel per unit time”.

Among these latencies, the latency caused by transfer rate restrictions can be reduced by decreasing the amount of data to be transferred itself, but the reality is that the other latency cannot be reduced.

The present disclosure has been made in view of such circumstances, and, in particular, reduces the latency of an event signal detection sensor.

A light detection device and a program according to an aspect of the present disclosure are a light detection device and a program including an event detecting section configured to output event data representing occurrence of an event that is a change in an electric signal, the event being detected in an event detection pixel configured to generate the electric signal by performing photoelectric conversion, a plurality of the event detection pixels being disposed on a two-dimensional plane; a received light amount calculation section configured to calculate a received light amount of the event detection pixel; and a latency correction section configured to correct information related to latency in the event data on the basis of the received light amount of the event detection pixel calculated by the received light amount calculation section.

A light detection method according to an aspect of the present disclosure is a light detection method including outputting event data representing occurrence of an event that is a change in an electric signal, the event being detected in an event detection pixel configured to generate the electric signal by performing photoelectric conversion, a plurality of the event detection pixels being disposed on a two-dimensional plane; calculating a received light amount of the event detection pixel; and correcting information related to latency in the event data on the basis of the calculated received light amount of the event detection pixel.

In an aspect of the present disclosure, event data representing occurrence of an event that is a change in an electric signal, the event being detected in an event detection pixel configured to generate the electric signal by performing photoelectric conversion, a plurality of the event detection pixels being disposed on a two-dimensional plane, is output, a received light amount of the event detection pixel is calculated, and information related to latency in the event data is corrected on the basis of the calculated received light amount of the event detection pixel.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that, in the present specification and the drawings, constituent elements having substantially the same functional configuration are denoted by the same reference signs, and redundant description thereof will be omitted.

1. Overview of Present Disclosure 2. First Embodiment 3. Second Embodiment 4. Third Embodiment 5. Fourth Embodiment 6. Fifth Embodiment 7. Application Example 8. Application Example to Mobile Body Hereinafter, modes for carrying out the present technology will be described. Description will be made in the following order.

The present disclosure, in particular, reduces latency in an event signal detection sensor.

1 FIG. First, the event signal detection sensor will be described.is a diagram depicting a configuration example of a signal processing device including an event signal detection sensor.

1 11 12 1 FIG. A signal processing deviceinincludes an event signal detection sensorand a subsequent-stage application.

11 21 31 11 31 12 The event signal detection sensoris composed of a so-called pixel array in which a plurality of pixel circuits, each including a pixelthat photoelectrically converts incident light to generate an electric signal, are disposed in a lattice shape on a two-dimensional plane. The event signal detection sensorgenerates, in the pixel, the electric signal by photoelectrically converting incident light, generates event data indicating occurrence of an event that is a change in the electric signal, and outputs the event data to the subsequent-stage application.

12 11 The subsequent-stage applicationis an application program that executes various types of processing on the basis of the event data supplied from the event signal detection sensor.

11 21 Here, in the event signal detection sensor, a portion in which the plurality of pixel circuitsare disposed is, as a whole, a portion that receives incident light and performs photoelectric conversion, and thus is also referred to as a light-receiving section.

2 FIG. 1 FIG. 21 Configuration Example of Pixel Circuitis a block diagram depicting a configuration example of the pixel circuitin.

21 31 32 The pixel circuitis constituted of the pixeland an event detecting section.

31 51 31 51 51 The pixelincludes a photodiode (PD)as a photoelectric conversion element. The pixel, in the PD, receives light incident on the PD, performs photoelectric conversion, and generates and produces a flow of a photocurrent (Iph) as an electric signal.

31 32 32 In a case in which a change exceeding a predetermined threshold (including a change equal to or greater than the threshold, as necessary) occurs in the photocurrent generated by the photoelectric conversion of the pixel, the event detecting sectiondetects the change in the photocurrent as an event. The event detecting sectionoutputs event data in response to (detection of) the event.

31 31 31 Here, a change in the photocurrent generated in the pixelcan be regarded as a light amount change in the light incident on the pixel, and thus the event can be regarded as a light amount change in the pixel(light amount change exceeding the threshold).

31 21 In the event data, at least position information (coordinates or the like) indicating a position of the pixel(pixel circuit) in which a light amount change occurred as the event can be specified. In addition, the event data includes information identifying a positive event and a negative event in accordance with a polarity (positive or negative) of the light amount change.

32 32 32 For a series of event data output by the event detecting sectionat the timing of event detection, time information representing a (relative) time at which the event occurred can be specified as long as an interval between the event data at the time of event occurrence is maintained as is. However, when the interval between the event data at the time of event occurrence is not maintained as it is due to the event data being stored in memory or the like, the time information is lost. Therefore, with regard to the event data, time information indicating a (relative) time at which the event occurred, such as a time stamp, is added to the event data before the interval between the event data at the time of event occurrence is no longer maintained as is. The processing of adding the time information to the event data may be performed by the event detecting sectionor may be performed outside of the event detecting sectionas long as before the interval between the event data at the time of event occurrence is no longer maintained as is.

32 41 42 43 The event detecting sectionincludes a current-to-voltage conversion section, a difference detecting section, and a comparator.

41 31 42 The current-to-voltage conversion sectionconverts the photocurrent from the pixelinto a voltage (hereinafter also referred to as photovoltage) Vo corresponding to a logarithm of the photocurrent, and outputs the photovoltage Vo to the difference detecting section.

42 41 43 The difference detecting sectioncalculates, for the photovoltage Vo from the current-to-voltage conversion section, a difference between a current photovoltage and a photovoltage at a timing different from the current timing by a very brief time, and outputs a difference signal Vout thereof to the comparator.

43 42 43 The comparatorcompares the difference signal Vout output from the difference detecting sectionand predetermined thresholds (voltages) +Vth and −Vth used for event detection. In a case in which the difference signal Vout is equal to or greater than the threshold +Vth or equal to or less than the threshold-Vth, the comparatordetermines that a light amount change as an event was detected (occurred), and outputs event data.

43 43 For example, in a case in which the difference signal Vout is equal to or greater than the threshold +Vth, the comparatordetermines that an event having positive polarity was detected and outputs event data having a positive event. On the other hand, in a case in which the difference signal Vout is equal to or less than the threshold −Vth, the comparatordetermines that a negative event was detected and outputs event data having a negative event.

11 31 The event signal detection sensor, as described above, detects a change in the amount of light received by the pixelas an event, and outputs, as event data, the coordinates of the pixel at that time, the time, and information indicating a positive event or a negative event.

11 In a case in which the event signal detection sensoroutputs the event data, when latency occurs with respect to the timing at which the event actually occurred, a delay occurs in the timing at which the event is detected, and the time information about the event data results in a time delayed from the time at which the event actually occurred.

3 FIG. 1 2 For example, as depicted in the left portion in, a case is considered in which a region where a white object exists on a gray background moves from a region WZindicated by a dotted line to a region WZin a left direction as indicated by an arrow.

2 2 3 FIG. In this case, at a boundary BZthat is a left end portion of the region WZof the white object, gray changes to white, thereby changing the received light amount such that the received light amount rises as depicted in an upper center portion in. This results in detection of a positive event.

1 2 3 FIG. On the other hand, at a boundary BZthat is a right end portion of the region WZof the white object, white changes to gray, thereby changing the received light amount such that the received light amount falls as depicted in an upper right portion in. This results in detection of a negative event.

1 2 The negative event detected at the boundary BZand the positive event detected at the boundary BZare detected at the same time. However, due to the occurrence of a delay (latency), information about a time delayed from the time at which the event actually occurred is recorded in the event data.

3 FIG. 11 12 1 That is, even if the positive event and the negative event depicted in the upper center portion and the upper right portion inactually occur at time t, time tdelayed by latency TLwhich is a predetermined delay time, for example, is recorded as the information about the time of the event data.

1 1 2 2 1 4 FIG. 4 FIG. 4 FIG. As a result, in a case of movement from the region WZwhere a white object is present at time Tas depicted in the topmost portion into the region WZat time Tas depicted in the second portion from the top in, a positive event is detected in a region Zesuch as depicted in the third portion from the top inwhen latency does not exist and the time at which the event actually occurs is appropriately reflected.

2 1 1 4 FIG. However, due to the effects of latency, time information delayed from the time at which the actual event occurred is recorded, and thus the region in which the positive event is detected becomes a past region Zeshifted in a right direction in the drawing by a distance Δd according to the latency TLwith respect to the actual region Ze, as depicted in bottommost portion in.

1 Therefore, conceivably, a correction is made by subtracting an amount equivalent to the latency Tthat is a predetermined delay time from the time information about the event data.

However, latency is known to be shorter as the received light amount increases, and, conversely, longer as the received light amount decreases. Therefore, subtracting, as a specific delay time, the delay time caused by latency from the time information does not necessarily result in an appropriate correction.

3 FIG. 5 FIG. 31 32 31 11 1 That is, in a case in which a positive event occurs in a state in which the received light amount is large as compared with the case in, for example, when the positive event actually occurs at time tas depicted in the left portion in, the time information about the event data is time tobtained by delaying the time tby latency TL, which is shorter than the latency TL.

3 FIG. 5 FIG. 41 42 41 12 1 In contrast, in a case in which a positive event occurs in a state in which the received light amount is small as compared with the case in, for example, when the positive event actually occurs at time tas depicted in the right portion in, the time information about the event data is time tobtained by delaying the time tby latency TL, which is longer than the latency TL.

2 1 2 2 1 2 6 FIG. 6 FIG. Accordingly, for example, latency LL generated in a case in which a foreground region LGof a light gray color moves on a background region LGindicated in a gray color darker than that of the foreground region LGand having a relatively high received light amount as depicted in the upper left portion inis smaller than latency DL generated in a case in which a foreground region DGof a dark gray color moves on a background region DGindicted in a gray color darker than that of the foreground region DGand having a relatively low received light amount as depicted in the upper right portion in.

That is, a length of the latency changes in accordance with the received light amount, making it impossible to appropriately reduce the effects caused by the latency by subtracting a specific time from the time information.

Here, in the present disclosure, the received light amount detected by the event signal detection sensor is detected, and the time information in the event data is corrected by the latency corresponding to the detected received light amount.

Thus, latency can be corrected in accordance with a length corresponding to the received light amount, and the effects caused by the latency are appropriately reduced in accordance with the received light amount.

1 2 4 FIG. This makes it possible to, for example, achieve correction such that the region where the positive event is detected approaches the region Zafrom the region Zain, and thus appropriately reduce the effects caused by the latency in accordance with the received light amount.

7 FIG. 7 FIG. 101 111 112 113 114 Next, a configuration example of the signal processing device of the present disclosure will be described with reference to. A signal processing deviceinincludes an event signal detection sensor, a visible light sensor, a subsequent-stage image sensor processor (ISP)/application processor (AP), and a subsequent-stage application.

111 11 113 1 FIG. The event signal detection sensorhas the same configuration as that of the event signal detection sensorin, detects a positive event or a negative event in accordance with a change in the received light amount, and outputs event data corresponding to the detection result to the subsequent-stage ISP/AP.

112 111 113 The visible light sensoris a sensor that detects the received light amount of the visible light received by the event signal detection sensor, and outputs a sensor output signal corresponding to the detected received light amount to the subsequent-stage ISP/AP.

8 FIG. 112 1 141 111 111 1 More specifically, as depicted in, the visible light sensoris not disposed on an optical axis AXof a lensprovided in a preceding stage of the event signal detection sensorand configured to collect light incident on the event signal detection sensor, but is disposed relatively close to the optical axis AX.

112 111 113 112 1 111 With such an arrangement, the visible light sensordetects a received light amount that can be regarded as being approximate to the received light amount of the event signal detection sensor, and outputs the received light amount to the subsequent-stage ISP/APas a sensor output signal. The visible light sensordetects the sensor output signal indicating the received light amount at one point in the vicinity the optical axis AX, and thus the received light amount calculated on the basis of the sensor output signal is calculated under the assumption that all pixels included in the event signal detection sensorreceive the same amount of received light.

113 111 112 The subsequent-stage ISP/APincludes a processor consisting of hardware, calculates the received light amount of the event signal detection sensoron the basis of the sensor output signal from the visible light sensor, and specifies the latency correction amount of the event data in accordance with the calculated received light amount.

113 111 114 Then, the subsequent-stage ISP/APcorrects the latency of the event data from the event signal detection sensorby the specified latency correction amount, and outputs the corrected event data to the subsequent-stage application.

113 112 114 At this time, the subsequent-stage ISP/APoutputs the sensor output signal supplied from the visible light sensorto the subsequent-stage application.

113 121 122 More specifically, the subsequent-stage ISP/APincludes an event detection pixel received light amount calculation sectionand a latency correction section.

121 31 111 122 2 FIG. The event detection pixel received light amount calculation sectioncalculates a received light amount per unit time of an event detection pixel (corresponding to the pixelin) constituting the event signal detection sensoron the basis of the sensor output signal, and outputs the received light amount to the latency correction section.

121 111 111 The event detection pixel received light amount calculation sectioncalculates, as the received light amount per unit time of the event detection pixel constituting the event signal detection sensor, a photon quantity per unit time incident on one event detection pixel constituting the event signal detection sensor, an irradiance [W/m2], or a luminance [Lux], for example, on the basis of the sensor output signal.

7 FIG. 8 FIG. 112 112 112 111 Note that, althoughanddepict an example in which one visible light sensoris provided for one event signal detection sensor, a configuration may be adopted in which a plurality of visible light sensors are provided for one event signal detection sensor. In a case in which a plurality of visible light sensorsare provided for one event signal detection sensor, the received light amount for each pixel may be calculated by using interpolation based on a positional relationship between each of the plurality of visible light sensorsand the event signal detection sensor.

122 111 121 The latency correction sectionspecifies the latency on the basis of the received light amount per unit time of the pixel constituting the event signal detection sensorsupplied from the event detection pixel received light amount calculation section, corrects a time stamp that is time information included in the event data by using the specified latency as a correction amount (hereinafter, also referred to as latency correction amount), and outputs the event data with the corrected time stamp as corrected event data.

111 122 111 Note that, in the present embodiment, the event signal detection sensoradds the information about a time stamp, which is time information, to the event data when generating the event data, and then outputs the result, but the latency correction sectionmay add a time stamp to the event data output from the event signal detection sensor, and may further add a correction corresponding to the latency.

122 121 9 FIG. More specifically, the latency correction sectionstores a table registering the latency for each event corresponding to the received light amount as shown in, for example, reads out the latency in the vicinity of the received light amount supplied from the event detection pixel received light amount calculation section, and calculates the corresponding latency by interpolation.

9 FIG. In the table in, the received light amount, the latency in rising edge detection which is a positive event, and the latency in falling edge detection which is a negative event are registered in this order from the left.

9 FIG. In, in a case in which the received light amount is 1000, it is registered that the latency is A seconds when the event is positive and that the latency is W seconds when the event is negative.

Further, in a case in which the received light amount is 100, it is registered that the latency is B seconds when the event is positive and that the latency is X seconds when the event is negative.

Furthermore, in a case in which the received light amount is 10, it is registered that the latency is C seconds when the event is positive, and the latency is Y seconds when the event is negative.

Further, in a case in which the received light amount is 1, it is registered that the latency is D seconds when the event is positive, and the latency is Z seconds when the event is negative.

9 FIG. A table such as shown inregisters the latencies determined for positive events and negative events in correspondence with received light amounts through experiments, simulations, and the like conducted in advance.

122 For example, in a case in which the received light amount is 500 and the event is positive, the latency correction sectionreads out, from the table, A seconds of the positive event at the received light amount of 1000 and B seconds of the positive event at the received light amount of 100, which are in the vicinity of the received light amount of 500, calculates the latency when the received light amount is 500 by interpolation by using, for example, a logarithmic average or the like, sets the calculation result as the latency correction amount, and corrects the time stamp that is the time information by the latency correction amount.

114 The subsequent-stage applicationconsists of software such as an application program, and executes signal processing based on the event data after the latency is corrected and the sensor output signal.

101 112 7 FIG. In the signal processing devicein, as described above, the sensor output signal according to the received light amount is output from the visible light sensor, the received light amount of the event detection pixel is calculated on the basis of the sensor output signal, and, on the basis of the calculated received light amount of the event detection pixel, the time stamp is corrected on the basis of the latency correction amount corresponding to the received light amount, which is measured in advance through experiments or the like.

111 As a result, the latency correction amount corresponding to the received light amount of the event signal detection sensoris specified, and the time stamp that is the time information is corrected by the specified latency correction amount, making it possible to appropriately reduce the effects caused by the latency.

101 7 FIG. 10 FIG. Next, the signal processing by the signal processing deviceinwill be described with reference to a flowchart in.

31 111 31 32 In step S, the event signal detection sensordetermines whether an event has been detected. In a case in which the determination is made in step Sthat an event has been detected, the processing proceeds to step S.

32 111 122 113 In step S, the event signal detection sensoroutputs the event data to the latency correction sectionof the subsequent-stage ISP/AP.

33 112 121 113 114 In step S, the visible light sensoroutputs a sensor output signal corresponding to the received light amount of visible light to the event detection pixel received light amount calculation sectionof the subsequent-stage ISP/AP. At this time, the sensor output signal is also output to the subsequent-stage application.

34 121 111 122 In step S, the event detection pixel received light amount calculation sectioncalculates the received light amount per unit time of the event detection pixel constituting the event signal detection sensoron the basis of the sensor output signal corresponding to the received light amount of visible light, and outputs the received light amount, which is the calculation result, to the latency correction section.

35 122 In step S, the latency correction sectionreads out the latency registered in association with the received light amount from the stored table, and specifies the latency correction amount by performing interpolation or the like, as necessary.

36 122 114 In step S, the latency correction sectioncorrects the time stamp, which is the time information, registered in the event data, by the latency correction amount, and outputs the result to the subsequent-stage application.

37 114 In step S, the subsequent-stage applicationexecutes signal processing based on the corrected event data and the sensor output signal, and outputs the execution result.

31 32 37 Further, in a case in which an event has not been detected in step S, the processing of steps Sto Sis skipped.

38 31 In step S, the determination is made as to whether termination of the processing has been instructed, and, in a case in which termination has not been instructed, the processing returns to step S, and the subsequent processing is repeated.

38 Then, in step S, in a case in which termination of the processing has been instructed, the processing ends.

Through the processing described above, the latency correction amount is specified in accordance with the received light amount, and the time stamp that is the time information about the event data is corrected by the specified latency correction amount.

As a result, the time information is corrected by setting an appropriate latency correction amount for the latency that changes in accordance with the received light amount, making it possible to appropriately reduce the effects of the latency.

112 111 111 112 In the above, an example has been described in which the visible light sensoris provided in the vicinity of the event signal detection sensor, the received light amount of the event signal detection sensoris calculated on the basis of the sensor output signal of the visible light sensor, the latency correction amount is determined in accordance with the calculated received light amount, and the time stamp that is time information is corrected by the latency correction amount according to the received light amount.

111 111 However, the received light amount varies in units of pixels constituting the event signal detection sensor, and thus the latency of the event data varies in units of pixels constituting the event signal detection sensor.

112 111 Therefore, a complementary metal oxide semiconductor (CMOS) image sensor may be provided instead of the visible light sensor, the received light amount of each pixel in the event signal detection sensormay be specified from a gradation signal of each pixel, and the latency may be corrected in units of pixels.

11 FIG. 112 111 depicts a configuration example of a signal processing device in which a CMOS image sensor is provided instead of the visible light sensor, the received light amount of each pixel in the event signal detection sensoris specified from the gradation signal of each pixel, and the latency is corrected in units of pixels.

101 101 101 101 11 FIG. 7 FIG. 11 FIG. 7 FIG. In a signal processing deviceA in, components having the same functions as those of the signal processing deviceinare denoted using the same reference signs, and the description thereof will be omitted as appropriate. Further, in the signal processing deviceA in, components having functions similar to those of the signal processing deviceinare denoted using the same reference signs followed by “A”.

101 101 151 112 113 113 11 FIG. 7 FIG. The signal processing deviceA indiffers from the signal processing deviceinin that a CMOS image sensoris provided instead of the visible light sensor, and a subsequent-stage ISP/APA is provided instead of the subsequent-stage ISP/AP.

151 113 The CMOS image sensorincludes a plurality of pixels disposed in an array in a direction perpendicular to an incident direction of incident light, and outputs a gradation signal corresponding to an amount of incident light to the subsequent-stage ISP/APon a pixel-by-pixel basis.

12 FIG. 151 141 2 111 141 1 111 More specifically, as depicted in, the CMOS image sensoris provided with a lensA-that condenses incident light in a preceding stage thereof. Further, at a front end of the event signal detection sensoras well, a lensA-that condenses incident light incident on the event signal detection sensoris provided.

11 12 141 1 141 2 151 111 151 111 Optical axes AXand AXof the lensesA-andA-are not identical, but are disposed relatively close and parallel to each other. Therefore, although parallax occurs between each pixel of the CMOS image sensorand each pixel of the event signal detection sensorwith respect to an object within a range relatively close to both, the gradation signal in each pixel of the CMOS image sensorcan be considered to generally correspond to the received light amount of each pixel of the event signal detection sensor.

151 113 111 151 111 Therefore, the CMOS image sensoroutputs the gradation signal in units of pixels to the subsequent-stage ISP/APA as information for detecting the received light amount that can be regarded as being approximate to the received light amount of each pixel of the event signal detection sensor. In other words, the gradation signal indicating the received light amount output from the CMOS image sensoris output as information indicating the received light amount of the pixel at the corresponding position configured in the event signal detection sensor.

113 113 111 111 151 7 FIG. The subsequent-stage ISP/APA has the same basic functions as those of the subsequent-stage ISP/APin, but calculates the received light amount in units of pixels in the event signal detection sensoron the basis of the event data from the event signal detection sensorand the gradation signal from the CMOS image sensor, and specifies the latency correction amount in units of pixels according to the calculated received light amount.

113 114 The subsequent-stage ISP/APA corrects the latency of the event data with the latency correction amounts specified per pixel, and outputs the event data corrected in units of pixels to the subsequent-stage application.

113 112 114 At this time, the subsequent-stage ISP/APA outputs the sensor output signal supplied from the visible light sensorto the subsequent-stage application.

113 121 122 More specifically, the subsequent-stage ISP/APA includes an event detection pixel received light amount calculation sectionA and a latency correction sectionA.

121 121 111 122 The event detection pixel received light amount calculation sectionA has the same functions as those of the event detection pixel received light amount calculation section, but calculates the received light amount per unit time for each event detection pixel constituting the event signal detection sensoron the basis of the gradation signal in units of pixels, and outputs the calculated received light amount to the latency correction sectionA.

121 111 151 The event detection pixel received light amount calculation sectionA calculates the received light amount per unit time for each pixel of the event detection pixels constituting the event signal detection sensoron the basis of the gradation signal of each pixel of the CMOS image sensor.

2 For example, in a case in which the received light amount per unit time for each pixel is determined as irradiance [W/m], the calculation is performed by the procedure below.

First, a charge amount per unit time of each pixel is determined by equation (1) below.

151 Here, Qu is a charge amount per unit time in units of pixels, and Q_after-gain is a charge amount after the gain is applied in units of pixels of the CMOS image sensorand is expressed by equation (2) below. Further, gain is a value of the applied gain and is expressed by equation (3) below.

64 Here, Vs is the value of the gradation signal, OPB is the gradation signal (generally) of a light-shielding pixel called optical black, Qs is a saturation charge amount of the pixel, and Vg is the gain value [dB]. In addition, “{circumflex over ( )}” represents exponentiation.

121 2 Then, in a case in which the spectrum of the incident light is known, the event detection pixel received light amount calculation sectionA calculates the received light amount per unit time for each pixel as the irradiance [W/m] by calculation with equation (4) below on the basis of the determined charge amount per unit time for each pixel.

2 2] Here, IRR is the irradiance [W/m] of the received light amount per unit time for each pixel, Qu1 is a charge amount generated in a pixel per second in units of pixels, and Qg is a charge amount per second that can be generated by photoelectric conversion of light having an irradiance of 1 [W/m.

2 Note that the charge amount Og per second that can be generated by the photoelectric conversion of light having an irradiance of 1 [W/m] needs to be determined in advance through experiments or theoretical calculations in accordance with the spectrum.

121 151 111 Then, the event detection pixel received light amount calculation sectionA, upon determination of the received light amount per unit time for each pixel of the CMOS image sensor, converts the coordinates to those of the corresponding pixel position on the event signal detection sensorfor each pixel, and sets the received light amount as the received light amount of the corresponding pixel.

121 151 111 13 FIG. More specifically, the event detection pixel received light amount calculation sectionA, upon determination of the received light amount per unit time of a pixel PC of the CMOS image sensor, converts the coordinates to those of a corresponding pixel PE on the event signal detection sensor, and applies the received light amount per unit time as the received light amount of the pixel PE obtained by the coordinate conversion, as depicted in.

151 121 111 Further, in the above, the assumption is made that the spectrum is known. However, in a case in which the CMOS image sensoris constituted of RGB pixels, the event detection pixel received light amount calculation sectionA may predict, in units of pixels of the event signal detection sensor, the spectrum from a ratio of a charge amount of each RGB pixel, and calculate the received light amount per unit time for each pixel by the calculation of equations (1) to (4) described above on the basis of the prediction result of the spectrum for each pixel.

14 FIG. For example, in a case in which the charge amount of the B pixel is greater than the charge amounts of the G pixel and the R pixel, there is a high possibility that the spectrum has a peak in a short wavelength region and thus, among waveforms shown in, a spectrum distribution such as indicated by a solid line is predicted, for example.

14 FIG. Further, for example, in a case in which the charge amount of the R pixel is greater than the charge amounts of the G pixel and the B pixel, there is a high possibility that the spectrum has a peak in a long wavelength region and thus, among waveforms shown in, a spectrum distribution as indicated by a dashed line is predicted, for example.

121 14 FIG. The event detection pixel received light amount calculation sectionA may estimate a spectrum from the ratio of the charge amounts of the RGB pixels by a method such as described with reference to.

14 FIG. Note that, in, the vertical axis represents intensity, the horizontal axis represents wavelength, and waveform examples of spectral distributions represented by the dashed line and the solid line are depicted.

122 122 111 121 The latency correction sectionA has the same basic functions as those of the latency correction section, but specifies a latency correction amount for each pixel on the basis of the received light amount per unit time of each pixel constituting the event signal detection sensorsupplied from the event detection pixel received light amount calculation sectionA, corrects the time stamp that is time information included in the event data for each pixel in accordance with the specified latency correction amount for each pixel, and outputs the event data with the time stamp corrected for each pixel as the corrected event data.

122 122 That is, the latency correction sectioncorrects the time information about the event data with the latency correction amount common to all pixels, whereas the latency correction sectionA corrects the time information about the event data for each pixel with the latency correction amount for each pixel.

15 FIG. 15 FIG. 122 For example, as shown in the left portion in, in a case in which the event data indicates a time stamp that is time information, coordinate information representing a pixel position, and information about an event type that indicates a positive event (rising edge) or a negative event (falling edge) in this order from the left in units of pixels, the latency correction sectionA corrects the time information as shown in the right portion inwhen the latency is one second, for example.

15 FIG. Note that, in the event data in the left portion in, the topmost portion indicates that, at the time information 00:00:10, a rising edge that is a positive event is detected in a pixel at a coordinate position (x, y)=(100, 200). The second portion from the top indicates that, at the time information 00:00:10, a rising edge that is a positive event is detected in a pixel at a coordinate position (x, y)=(200, 200). The third portion from the top indicates that, at the time information 00:00:11, a rising edge that is a positive event is detected in a pixel at a coordinate position (x, y)=(100, 200). The fourth portion from the top indicates that, at the time information 00:00:11, a rising edge that is a positive event is detected in a pixel at a coordinate position (x, y)=(200, 200).

15 FIG. Further, in the event data in the right portion in, the topmost portion indicates that, at the time information 00:00:09, a rising edge that is a positive event is detected in a pixel at a coordinate position (x, y)=(100, 200). The second portion from the top indicates that, at the time information 00:00:09, a rising edge that is a positive event is detected in a pixel at a coordinate position (x, y)=(200, 200). The third portion from the top indicates that, at the time information 00:00:10, a rising edge that is a positive event is detected in a pixel at a coordinate position (x, y)=(100, 200). The fourth portion from the top indicates that, at the time information 00:00:10, a rising edge that is a positive event is detected in a pixel at a coordinate position (x, y)=(200, 200).

15 FIG. 15 FIG. That is, the time information in the original event data shown in the left portion inis corrected in the right portion inby subtracting one second, which is the latency correction amount.

Further, in the above, an example has been described in which, among the event data, the time information is corrected by the latency correction amount. However, the corrected time information may be added.

16 FIG. 16 FIG. For example, as shown in the right portion in, the corrected time information may be added to event data such as shown in the left portion in.

16 FIG. 15 FIG. 16 FIG. 15 FIG. The left portion inis the same as the left portion in, and the right portion inhas a corrected time information column additionally recorded between the time information column and the coordinate information column of the right portion in.

16 FIG. Note that the latency correction amount differs in units of pixels. Thus, a column in which a delay time corresponding to latency, that is, the latency correction amount, is recorded may be used instead of the column of the corrected time information in the right portion in, and time information obtained by correcting the original time information by the latency correction amount may be actually used.

101 11 FIG. 17 FIG. Next, the signal processing by the signal processing deviceA inwill be described with reference to a flowchart in.

51 111 51 52 In step S, the event signal detection sensordetermines whether an event has been detected. In a case in which the determination is made in step Sthat an event has been detected, the processing proceeds to step S.

52 111 122 113 In step S, the event signal detection sensoroutputs the event data to the latency correction sectionA of the subsequent-stage ISP/APA.

53 151 121 113 114 In step S, the CMOS image sensoroutputs the gradation signal of each pixel to the event detection pixel received light amount calculation sectionA of the subsequent-stage ISP/APA. At this time, the gradation signal is also output to the subsequent-stage application.

54 121 111 122 In step S, the event detection pixel received light amount calculation sectionA calculates the received light amount per unit time in units of pixels of the event detection pixel constituting the event signal detection sensoron the basis of the gradation signal in units of pixels, and outputs the received light amount for each pixel that is the calculation result to the latency correction sectionA.

55 122 In step S, the latency correction sectionA reads out, for each pixel, the latency registered in association with the received light amount from the stored table, and specifies the latency correction amount for each pixel by performing interpolation or the like, as necessary.

56 122 114 In step S, the latency correction sectionA corrects, for each pixel, the time stamp, which is the time information registered in the event data, by the latency correction amount, and outputs the result to the subsequent-stage application.

57 114 In step S, the subsequent-stage applicationexecutes signal processing based on the corrected event data and the gradation signal, and outputs the execution result.

51 52 57 Further, in a case in which an event has not been detected in step S, the processing of steps Sto Sis skipped.

58 51 In step S, the determination is made as to whether termination of the processing has been instructed, and, in a case in which termination has not been instructed, the processing returns to step S, and the subsequent processing is repeated.

58 Then, in step S, in a case in which termination of the processing has been instructed, the processing ends.

Through the processing described above, the latency correction amount for each pixel is specified in accordance with the received light amount for each pixel, and the time stamp that is the time information about the event data is corrected for each pixel by the specified latency correction amount for each pixel.

As a result, the time information is corrected in units of pixels by setting an appropriate latency correction amount for the latency that changes in accordance with the received light amount in units of pixels, making it possible to appropriately reduce the effects of the latency.

114 In the above, an example has been described in which the time stamp that is the time information in the event data is corrected by the latency correction amount corresponding to the received light amount. However, only the information about the received light amount may be added to the event data, and the latency corresponding to the received light amount may be corrected in the subsequent-stage applicationthat has acquired the event data.

18 FIG. 18 FIG. 114 114 That is, for example, as shown in the right portion in, information about the received light amount is added to original event data such as shown in the left portion in, and the result is output to the subsequent-stage application. In the subsequent-stage application, the latency is corrected on the basis of the event data to which the received light amount is added, and the processing is subsequently executed.

18 FIG. 15 FIG. 16 FIG. 18 FIG. 18 FIG. Note that the left portion inis the original event data, similar to the left portions inand, while the right portion inhas a received light amount column for the corresponding pixels provided between the time stamp column indicating the time information and the coordinate information column indicating pixel positions in the left portion in.

18 1111 1111 1112 1112 FIGS.,,,, and In the received light amount column in the right portion inare registered as the received light amounts in this order from the top.

18 FIG. That is, the topmost portion in the right portion inindicates that, at the time information 00:00:10, a rising edge that is a positive event is detected in a pixel at a coordinate position (x, y)=(100, 200), and the received light amount is 1111. The second portion from the top indicates that, at the time information 00:00:10, a rising edge that is a positive event is detected in a pixel at a coordinate position (x, y)=(200, 200), and the received light amount is 1111. The third portion from the top indicates that, at the time information 00:00:11, a rising edge that is a positive event is detected in a pixel at a coordinate position (x, y)=(100, 200), and the received light amount is 1112. The fourth portion from the top indicates that, at the time information 00:00:11, a rising edge that is a positive event is detected in a pixel at a coordinate position (x, y)=(200, 200), and the received light amount is 1112.

114 The subsequent-stage applicationdetermines a latency correction amount corresponding to the received light amount on the basis of the event data corrected by the application of the information about the received light amount, corrects the time stamp that is the time information, and executes the processing.

19 FIG. depicts a configuration example of a signal processing device configured to register the received light amount in the event data.

101 101 101 101 101 101 19 FIG. 7 FIG. 11 FIG. 19 FIG. 7 FIG. 11 FIG. In a signal processing deviceB in, components having the same functions as those of the signal processing deviceinand the signal processing deviceA inare denoted using the same reference signs, and the description thereof will be omitted as appropriate. Further, in the signal processing deviceB in, components having functions similar to those of the signal processing deviceinand the signal processing deviceA inare denoted using the same reference signs followed by “B”.

101 101 101 113 114 113 113 114 19 FIG. 7 FIG. 11 FIG. The signal processing deviceB indiffers from the signal processing deviceinand the signal processing deviceA inin that a subsequent-stage ISP/APB and a subsequent-stage applicationB are provided instead of the subsequent-stage ISP/APsandA and the subsequent-stage application.

113 121 122 More specifically, the subsequent-stage ISP/APB includes an event detection pixel received light amount calculation sectionB and a latency correction sectionB.

121 121 111 122 The event detection pixel received light amount calculation sectionB has the same functions as those of the event detection pixel received light amount calculation sectionA, calculates the received light amount per unit time for each event detection pixel constituting the event signal detection sensor, and outputs the received light amounts to the latency correction sectionB.

122 111 114 18 FIG. The latency correction sectionB, as described with reference to, adds the received light amount for each event detection pixel to the event data supplied from the event signal detection sensor, and outputs the result to the subsequent-stage applicationB as the corrected event data.

122 122 122 That is, unlike the latency correction sectionsandA, the latency correction sectionB does not correct the time stamp, which is the time information included in the event data, and outputs the event data only by adding the information about the received light amount.

122 Note that the latency correction sectionB does not correct the time stamp that is the time information, but provides information about the received light amount that is required when the time stamp that is the time information about the event data is corrected, and thus achieves the function of correcting the event data.

114 114 The subsequent-stage applicationB has the same basic functions as those of the subsequent-stage application, but further corrects the time stamp that is time information by the latency correction amount on the basis of the event data with the information about the received light amount added thereto, and executes the processing.

114 171 171 9 FIG. More specifically, the subsequent-stage applicationB includes a latency correction section. The latency correction sectionincludes a table (similar to the table in) consisting of received light amounts and corresponding latencies (latency correction amounts), reads out the corresponding latency correction amount from the table on the basis of the information about the received light amount included in the event data, and corrects the time stamp that is the time information.

101 19 FIG. 20 FIG. Next, the signal processing by the signal processing deviceB inwill be described with reference to a flowchart in.

71 111 71 72 In step S, the event signal detection sensordetermines whether an event has been detected. In a case in which the determination is made in step Sthat an event has been detected, the processing proceeds to step S.

72 111 122 113 In step S, the event signal detection sensoroutputs the event data to the latency correction sectionB of the subsequent-stage ISP/APB.

73 151 121 113 114 In step S, the CMOS image sensoroutputs the gradation signal of each pixel to the event detection pixel received light amount calculation sectionB of the subsequent-stage ISP/APB. At this time, the gradation signal is also output to the subsequent-stage applicationB.

74 121 111 122 In step S, the event detection pixel received light amount calculation sectionB calculates the received light amount per unit time in units of pixels of the event detection pixels constituting the event signal detection sensoron the basis of the gradation signal in units of pixels, and outputs the received light amount for each pixel that is the calculation result to the latency correction sectionB.

75 122 In step S, the latency correction sectionB corrects the event data by adding the information about the received light amount to the event data for each pixel.

76 122 114 In step S, the latency correction sectionB outputs, to the subsequent-stage applicationB, the event data corrected by the addition of the information about the received light amount, for each pixel.

77 114 In step S, the subsequent-stage applicationB acquires the event data corrected by the addition of the information about the received light amount, for each pixel.

78 171 114 In step S, the latency correction sectionof the subsequent-stage applicationB reads out information about the received light amount for each pixel from the event data, and searches for and specifies a corresponding latency correction amount from the table in which latencies for received light amounts are recorded.

79 171 In step S, the latency correction sectioncorrects, in the event data, the time stamp that is time information by the latency correction amount for each pixel.

80 114 In step S, the subsequent-stage applicationB executes the signal processing based on the event data with the corrected time stamp that is time information and the gradation signal, and outputs the execution result.

71 72 80 Further, in a case in which an event has not been detected in step S, the processing of steps Sto Sis skipped.

81 71 In step S, the determination is made as to whether termination of the processing has been instructed, and, in a case in which termination has not been instructed, the processing returns to step S, and the subsequent processing is repeated.

81 Then, in step S, in a case in which termination of the processing has been instructed, the processing ends.

Through the processing described above as well, the latency correction amount for each pixel is specified in accordance with the received light amount for each pixel, and the time stamp that is the time information about the event data is corrected for each pixel by the specified latency correction amount.

As a result, the time information is corrected in units of pixels by setting an appropriate latency correction amount for the latency that changes in accordance with the received light amount in units of pixels, making it possible to appropriately reduce the effects of the latency.

101 101 101 151 111 151 111 In the above, configuration examples of the signal processing devices,A, andB that use the CMOS image sensorand the event signal detection sensorhave been described. However, a hybrid event-base vision sensor (EVS) having the functions of both the CMOS image sensorand the event signal detection sensormay be used.

21 FIG. 21 FIG. 21 FIG. 151 201 A hybrid EVS sensor has a pixel structure such as depicted in the right portion in, for example. Note that the left portion indepicts the pixel configuration of the CMOS image sensorand depicts differences in configuration from a hybrid EVSin the right portion in.

151 21 FIG. That is, the CMOS image sensorhas a pixel arrangement in which, as depicted in the left portion in, for example, 2 pixels×2 pixels are set as one pixel unit, and RGB pixel units are disposed in a Bayer array.

201 151 31 111 2 FIG. In contrast, the hybrid EVShas a configuration in which, in each of the four pixels that constitute the R pixel unit and the four pixels that constitute the B pixel unit in the CMOS image sensor, two vertically aligned pixels are replaced with the event detection pixel (corresponding to pixelin) EVS in the event signal detection sensor.

201 151 With such a configuration, in the hybrid EVS sensor, the gradation signal is acquired from the RGB pixels as in the configuration of the CMOS image sensor, and the event data is acquired from the event detection pixel EVS.

201 141 201 21 141 201 22 FIG. 21 FIG. Further, in the hybrid EVS sensor, a configuration is adopted in which a lensB for condensing light is provided at a front end as depicted in, and thus the gradation signal and the event data acquired in the hybrid EVScan be acquired in a state in which an appropriate positional relationship in which a parallax in arrangement or the like does not occur with respect to an axis AXof the lensB is maintained. Note that the pixel array of the hybrid EVSin the right portion inis an example, and may be a pixel array other than the illustrated pixel array.

23 FIG. 23 FIG. 101 201 114 First Configuration Example of Signal Processing Device Using Hybrid EVSdepicts a first configuration example of a signal processing device that uses a hybrid EVS sensor. A signal processing deviceC inis constituted of the hybrid EVS sensorand the subsequent-stage application.

201 221 222 223 224 The hybrid EVS sensorincludes an event data acquisition section, a gradation signal acquisition section, an event detection pixel received light amount calculation section, and a latency correction section.

221 224 21 FIG. The event data acquisition sectionacquires the event data output from the event detection pixels EVS inand outputs the event data to the latency correction section.

222 223 21 FIG. The gradation signal acquisition sectionacquires the gradation signals output from the RGB pixels in, and outputs the gradation signals to the event detection pixel received light amount calculation section.

221 111 222 151 The event data acquisition sectionfunctions similarly to the event signal detection sensor. Further, the gradation signal acquisition sectionfunctions similarly to the CMOS image sensor.

223 224 121 121 122 122 The event detection pixel received light amount calculation sectionand the latency correction sectionfunction as the event detection pixel received light amount calculation sectionA orB and the latency correction sectionA orB, respectively.

101 101 101 23 FIG. 11 FIG. 19 FIG. That is, the signal processing deviceC incan substantially function as the signal processing deviceA inor the signal processing deviceB in.

Note that the respective signal processing is the same, and thus description thereof will be omitted.

201 221 222 223 224 221 222 In the above, a configuration example has been described in which the hybrid EVS sensorincludes the event data acquisition section, the gradation signal acquisition section, the event detection pixel received light amount calculation section, and the latency correction section. However, a configuration in which only the event data acquisition sectionand the gradation signal acquisition sectionare provided may be adopted.

24 FIG. 221 222 201 depicts a configuration example of a signal processing device in a case in which only the event data acquisition sectionand the gradation signal acquisition sectionare provided in the hybrid EVS sensor.

101 201 241 114 24 FIG. A signal processing deviceD inis constituted of a hybrid EVS sensorD, a subsequent-stage ISP/AP, and the subsequent-stage application.

201 201 221 222 221 222 The hybrid EVS sensorD has the same basic functions as those of the hybrid EVS sensor, but is provided with only an event data acquisition sectionD and a gradation signal acquisition sectionD having the same functions as those of the event data acquisition sectionand the gradation signal acquisition section, respectively.

241 223 224 223 224 The subsequent-stage ISP/APis provided with an event detection pixel received light amount calculation sectionD and a latency correction sectionD having the same functions as those of the event detection pixel received light amount calculation sectionand the latency correction section, respectively.

101 201 201 241 24 FIG. 23 FIG. That is, in the signal processing deviceD in, the functions of the hybrid EVS sensorinare realized by the hybrid EVS sensorD and the subsequent-stage ISP/AP.

101 101 101 24 FIG. 11 FIG. 19 FIG. With such a configuration, the signal processing deviceD inalso substantially realizes functions as the signal processing deviceA inor the signal processing deviceB in.

201 221 222 223 In the above, an example has been described in which the hybrid EVS sensorD is constituted only by the event data acquisition sectionD and the gradation signal acquisition sectionD. However, the functions of the event detection pixel received light amount calculation sectionmay be further provided.

25 FIG. 201 221 222 223 depicts a configuration example of a signal processing device in a case in which the hybrid EVS sensoris realized by the event data acquisition section, the gradation signal acquisition section, and the event detection pixel received light amount calculation section.

101 201 241 114 25 FIG. A signal processing deviceE inis constituted of a hybrid EVS sensorE, a subsequent-stage ISP/APE, and the subsequent-stage application.

201 201 221 222 223 221 222 223 The hybrid EVS sensorE has the same basic functions as those of the hybrid EVS sensor, but is provided with an event data acquisition sectionE, a gradation signal acquisition sectionE, and an event detection pixel received light amount calculation sectionE having the same functions as those of the event data acquisition section, the gradation signal acquisition section, and the event detection pixel received light amount calculation section, respectively.

241 224 224 The subsequent-stage ISP/APE is provided with a latency correction sectionE having the same functions as those of the latency correction section.

101 201 201 241 101 101 101 25 FIG. 23 FIG. 25 FIG. 11 FIG. 19 FIG. That is, in the signal processing deviceE in, the functions of the hybrid EVS sensorinare realized by the hybrid EVS sensorE and the subsequent-stage ISP/APE. With such a configuration, the signal processing deviceE inalso substantially realizes the functions of the signal processing deviceA inor the signal processing deviceB in.

201 151 In the above, a configuration example of the signal processing device that uses the hybrid EVS sensorhas been described. However, the gradation signal and the event data may be switched and acquired from the pixel signal of the pixel array constituting the CMOS image sensor.

26 FIG. 151 depicts a configuration example of a signal processing device that switches between acquiring the gradation signal and acquiring the event data from the pixel signal of the pixel array constituting the CMOS image sensor.

101 261 113 114 26 FIG. A signal processing deviceF inincludes a switching sensor, the subsequent-stage ISP/APA, and the subsequent-stage application.

261 270 151 271 270 272 The switching sensorincludes a pixel arrayconstituting the CMOS image sensor, an event data detecting sectionthat acquires the event data on the basis of a signal output from the pixel array, and a gradation signal detecting sectionthat detects a gradation signal.

270 271 272 The signal output from the pixel arrayis switched between being output to only the event data detecting sectionand being output to only the gradation signal detecting section.

271 270 122 113 Therefore, the event data detecting sectiondetects event data at a timing when a signal is output from the pixel arrayto itself, and outputs the event data to the latency correction sectionA of the subsequent-stage ISP/AP.

272 270 121 113 Further, the gradation signal detecting sectiondetects a gradation signal at a timing when a signal is output from the pixel arrayto itself, and outputs the gradation signal to the event detection pixel received light amount calculation sectionA of the subsequent-stage ISP/AP.

261 151 111 That is, the switching sensorfunctions as a configuration having both the functions of the CMOS image sensorand the event signal detection sensordescribed above.

101 101 101 26 FIG. 11 FIG. 19 FIG. As a result, with such a configuration, the signal processing deviceF inalso substantially realizes the functions of the signal processing deviceA inor the signal processing deviceB in.

In the above, an example has been described in which the latency correction amount is set in accordance with the received light amount, and the time stamp that is the time information in the event data is corrected by the latency correction amount, thereby correcting the event data.

114 114 However, in a case in which the event data is transmitted to the subsequent-stage applicationin units of event data groups, each consisting of a plurality of event data, an event data group configured to be shifted by the latency correction amount specified in accordance with the received light amount may be output to the subsequent-stage application.

27 FIG. More specifically, as depicted in the upper portion in, each strip-shaped frame represents a single event data item, the data is assumed to be generated sequentially in the right direction in the drawing, and the effects of latency are assumed to occur for a duration equivalent to two event data items.

27 FIG. 1 1 14 114 Here, in the upper portion in, a case is considered in which, after a predetermined quantity of event data items generated in time series is sequentially buffered, an event data group Wconsisting of event data Dto Dis output to the subsequent-stage application, for example.

1 114 114 1 2 In this case, when the event data group Wis transmitted to the subsequent-stage applicationas is, the subsequent-stage applicationprocesses, as is, the event data Dand D, which are already past event data, at the timing of acquisition of the event data, resulting in the occurrence of the effects of the latency.

122 113 21 1 2 2 3 16 114 27 FIG. Therefore, when transmitting the event data group, the latency correction sectionof the subsequent-stage ISP/APmay shift the event data group by a latency correction amount SF, skipping the two pieces of event data Dand Dexpected to be affected by the latency, and transmit an event data group Wconsisting of the subsequent-stage event data Dto Dto the subsequent-stage application, as depicted in the lower portion in.

114 2 2 2 By this processing, the event data group is shifted and transmitted in a state in which the past event data is skipped by an amount corresponding to the latency. Accordingly, when the subsequent-stage applicationacquires the event data group W, the event data group is acquired as the event data group Wfrom which the effects of latency have been removed, and thus processing in which the effects of the latency are corrected is realized by simply processing the event data constituting the acquired event data group Wsequentially in time series.

28 FIG. 27 FIG. 7 FIG. 101 Next, the signal processing in the application example will be described with reference to a flowchart in. Note that, here, the processing in a case in which the signal processing with reference tois performed by the signal processing deviceinwill be described, and the assumption is made that all pixels have the same received light amount and the event data of all pixels is corrected by a unified latency correction amount.

101 111 101 102 In step S, the event signal detection sensordetermines whether an event has been detected. In a case in which the determination is made in step Sthat an event has been detected, the processing proceeds to step S.

102 111 122 113 122 In step S, the event signal detection sensoroutputs the event data to the latency correction sectionof the subsequent-stage ISP/AP. At this time, the latency correction sectionbuffers the event data.

103 111 101 In step S, the event signal detection sensordetermines whether the buffered event data quantity is equal to or greater than a predetermined threshold, and, in a case in which the quantity is not equal to or greater than the predetermined threshold, the processing returns to step S.

27 FIG. Note that the predetermined threshold here is, for example, a value obtained by adding the quantity of event data items corresponding to the assumed latency to the quantity of event data items transmitted as the event data group described with reference to.

27 FIG. Accordingly, in the example in, because the latency involves two event data items, the predetermined threshold is 16 or greater.

103 104 In step S, in a case in which the determination is made that the buffered event data quantity is equal to or greater than the predetermined threshold, the processing proceeds to step S.

104 112 121 113 In step S, the visible light sensoroutputs a sensor output signal corresponding to the received light amount of visible light to the event detection pixel received light amount calculation sectionof the subsequent-stage ISP/AP.

105 121 111 122 In step S, the event detection pixel received light amount calculation sectioncalculates the received light amount per unit time of the event detection pixels constituting the event signal detection sensoron the basis of the sensor output signal corresponding to the received light amount of the visible light, and outputs the received light amount that is the calculation result to the latency correction section.

106 122 In step S, the latency correction sectionreads out the latency registered in association with the received light amount from the stored table, and specifies the latency correction amount by performing interpolation or the like, as necessary.

107 122 114 In step S, the latency correction sectionshifts, in the buffered event data, event data by an amount corresponding to the latency correction amount, and then forms an event data group consisting of a predetermined quantity of event data items, thereby correcting the latency, and outputs the event data group to the subsequent-stage application.

108 114 In step S, the subsequent-stage applicationsequentially processes the event data constituting the event data group on the basis of the event data group in which the latency is corrected, thereby executing the signal processing, and outputs the execution result.

101 102 108 Further, in a case in which an event has not been detected in step S, the processing of steps Sto Sis skipped.

109 101 In step S, the determination is made as to whether termination of the processing has been instructed, and, in a case in which termination has not been instructed, the processing returns to step S, and the subsequent processing is repeated.

109 Then, in step S, in a case in which termination of the processing has been instructed, the processing ends.

114 With the above processing, the latency correction amount is specified in accordance with the received light amount, and the event data group is configured to be shifted by the specified latency correction amount, making it possible to realize, in the subsequent-stage application, processing in which the effects of latency are corrected by simply sequentially processing the event data constituting the acquired event data group in time series.

As a result, it is possible to appropriately reduce the effects of latency that changes in accordance with the received light amount.

27 FIG. 7 FIG. 11 FIG. 23 FIG. 24 FIG. 25 FIG. 26 FIG. 101 101 101 101 101 101 Note that, in the above, the processing in a case in which the signal processing referenced inis performed by the signal processing deviceinhas been described. However, similar processing can be realized in units of pixels in any of the signal processing deviceA in, the signal processing deviceC in, the signal processing deviceD in, the signal processing deviceE in, and the signal processing deviceF in.

114 In the above, an example has been described in which, among buffered event data, an event data group to be transmitted is shifted by the latency correction amount and output to the subsequent-stage application.

1 2 However, the possibility exists that the processing may be affected by the loss of event data Dand Dthat were set not to be transmitted, such as when a need arises to make reference thereto.

114 Therefore, for the initial event data group, the event data of the subsequent stage may be increased by the latency correction amount, and may be output to the subsequent-stage application.

1 14 1 114 11 1 16 15 16 29 FIG. 29 FIG. That is, for the event data Dto Dof the event data group Winitially intended to be transmitted to the subsequent-stage applicationas depicted in the upper portion in, an event data group Wconsisting of the event data Dto Dobtained by adding the event data Dand Dof a subsequent stage corresponding to the effect of the latency is configured to be transmitted as depicted in the lower portion in.

114 3 16 1 16 1 2 Then, the subsequent-stage applicationmay execute processing by selectively using the event data Dto Dthat excludes, from the event data group consisting of the event data Dto D, the past event data Dand Dthat are in the past by an amount of time corresponding to the latency.

29 FIG. 11 1 1 14 15 16 21 That is, as depicted in the lower portion in, the event data group W, obtained by increasing the event data group Wconsisting of the event data Dto Dbefore correction by the event data Dand Sequivalent to the shift amount SFcorresponding to the portion affected by the latency, is transmitted.

114 11 1 2 In this case, the subsequent-stage applicationcan recognize, among the event data at the head of the event data group W, the event data Dand Dcorresponding to the portion affected by the latency as past event data from the time stamps or the like, and thus executes processing while excluding this event data from the processing in advance, making it possible to realize processing not affected by the latency.

1 2 114 Further, as described above, in a case in which reference needs to be made to the past event data Dand Din accordance with processing contents or the like, the event data is transmitted as an event data group without loss, and thus can be referenced by the subsequent-stage application.

The event data group may be configured by deleting only the past event data affected by the latency.

30 FIG. 30 FIG. 21 1 1 2 21 That is, as depicted in the lower portion in, an event data group W, obtained by deleting, from the event data group Wbefore correction depicted in the upper portion in, only the event data Dand Dequivalent to the shift amount SFcorresponding to the portion affected by the latency, is transmitted.

114 21 11 In this case, the subsequent-stage applicationsequentially executes processing from the event data at the head of the event data group W, making it possible to realize processing not affected by the latency. In this case, although loss occurs in the chronologically later event data relative to the initial event data group W, only the event data corresponding to the latency amount is lost, minimizing the effect.

The technology according to the present disclosure (present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, and a robot.

31 FIG. is a block diagram depicting an example of a schematic configuration of a vehicle control system as an example of a mobile body control system to which the technology according to an embodiment of the present disclosure can be applied.

12000 12001 12000 12010 12020 12030 12040 12050 12051 12052 12053 12050 31 FIG. The vehicle control systemincludes a plurality of electronic control units connected to each other via a communication network. In the example depicted in, the vehicle control systemincludes a driving system control unit, a body system control unit, an outside-vehicle information detecting unit, an in-vehicle information detecting unit, and an integrated control unit. In addition, a microcomputer, a sound/image output section, and a vehicle-mounted network interface (I/F)are illustrated as a functional configuration of the integrated control unit.

12010 12010 The driving system control unitcontrols the operation of devices related to the driving system of the vehicle in accordance with various kinds of programs. For example, the driving system control unitfunctions as a control device for a driving force generating device for generating the driving force of the vehicle, such as an internal combustion engine, a driving motor, or the like, a driving force transmitting mechanism for transmitting the driving force to wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating the braking force of the vehicle, and the like.

12020 12020 12020 12020 The body system control unitcontrols the operation of various kinds of devices provided to a vehicle body in accordance with various kinds of programs. For example, the body system control unitfunctions as a control device for a keyless entry system, a smart key system, a power window device, or various kinds of lamps such as a headlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or the like. In this case, radio waves transmitted from a mobile device as an alternative to a key or signals of various kinds of switches can be input to the body system control unit. The body system control unitreceives these input radio waves or signals, and controls a door lock device, the power window device, the lamps, or the like of the vehicle.

12030 12000 12030 12031 12030 12031 12030 The outside-vehicle information detecting unitdetects information about the outside of the vehicle including the vehicle control system. For example, the outside-vehicle information detecting unitis connected with an imaging section. The outside-vehicle information detecting unitmakes the imaging sectionimage an image of the outside of the vehicle, and receives the imaged image. On the basis of the received image, the outside-vehicle information detecting unitmay perform processing of detecting an object such as a human, a vehicle, an obstacle, a sign, a character on a road surface, or the like, or processing of detecting a distance thereto.

12031 12031 12031 The imaging sectionis an optical sensor that receives light, and which outputs an electric signal corresponding to a received light amount of the light. The imaging sectioncan output the electric signal as an image, or can output the electric signal as information about a measured distance. In addition, the light received by the imaging sectionmay be visible light, or may be invisible light such as infrared rays or the like.

12040 12040 12041 12041 12041 12040 The in-vehicle information detecting unitdetects information about the inside of the vehicle. The in-vehicle information detecting unitis, for example, connected with a driver state detecting sectionthat detects the state of a driver. The driver state detecting section, for example, includes a camera that images the driver. On the basis of detection information input from the driver state detecting section, the in-vehicle information detecting unitmay calculate a degree of fatigue of the driver or a degree of concentration of the driver, or may determine whether the driver is dozing.

12051 12030 12040 12010 12051 The microcomputercan calculate a control target value for the driving force generating device, the steering mechanism, or the braking device on the basis of the information about the inside or outside of the vehicle which is obtained by the outside-vehicle information detecting unitor the in-vehicle information detecting unit, and output a control command to the driving system control unit. For example, the microcomputercan perform cooperative control intended to implement functions of an advanced driver assistance system (ADAS) which functions include collision avoidance or shock mitigation for the vehicle, following driving based on a following distance, vehicle speed maintaining driving, a warning of collision of the vehicle, a warning of deviation of the vehicle from a lane, or the like.

12051 12030 12040 In addition, the microcomputercan perform cooperative control intended for automated driving, which makes the vehicle travel automatedly without depending on the operation of the driver, or the like, by controlling the driving force generating device, the steering mechanism, the braking device, or the like on the basis of the information about the outside or inside of the vehicle which is obtained by the outside-vehicle information detecting unitor the in-vehicle information detecting unit.

12051 12030 12030 12051 12030 In addition, the microcomputercan output a control command to the body system control uniton the basis of the information about the outside of the vehicle which is obtained by the outside-vehicle information detecting unit. For example, the microcomputercan perform cooperative control intended to prevent a glare by controlling the headlamp so as to change from a high beam to a low beam, for example, in accordance with the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detecting unit.

12052 12061 12062 12063 12062 31 FIG. The sound/image output sectiontransmits an output signal of at least one of a sound and an image to an output device capable of visually or auditorily notifying information to an occupant of the vehicle or the outside of the vehicle. In the example of, an audio speaker, a display section, and an instrument panelare illustrated as the output device. The display sectionmay, for example, include at least one of an on-board display and a head-up display.

32 FIG. 12031 is a diagram depicting an example of the installation position of the imaging section.

32 FIG. 12031 12101 12102 12103 12104 12105 In, the imaging sectionincludes imaging sections,,,, and.

12101 12102 12103 12104 12105 12100 12101 12105 12100 12102 12103 12100 12104 12100 12105 The imaging sections,,,, andare, for example, disposed at positions on a front nose, sideview mirrors, a rear bumper, and a back door of the vehicleas well as a position on an upper portion of a windshield within the interior of the vehicle. The imaging sectionprovided to the front nose and the imaging sectionprovided to the upper portion of the windshield within the interior of the vehicle obtain mainly an image of the front of the vehicle. The imaging sectionsandprovided to the sideview mirrors obtain mainly an image of the sides of the vehicle. The imaging sectionprovided to the rear bumper or the back door obtains mainly an image of the rear of the vehicle. The imaging sectionprovided to the upper portion of the windshield within the interior of the vehicle is used mainly to detect a preceding vehicle, a pedestrian, an obstacle, a signal, a traffic sign, a lane, or the like.

32 FIG. 12101 12104 12111 12101 12112 12113 12102 12103 12114 12104 12100 12101 12104 Incidentally,depicts an example of photographing ranges of the imaging sectionsto. An imaging rangerepresents the imaging range of the imaging sectionprovided to the front nose. Imaging rangesandrespectively represent the imaging ranges of the imaging sectionsandprovided to the sideview mirrors. An imaging rangerepresents the imaging range of the imaging sectionprovided to the rear bumper or the back door. A bird's-eye image of the vehicleas viewed from above is obtained by superimposing image data imaged by the imaging sectionsto, for example.

12101 12104 12101 12104 At least one of the imaging sectionstomay have a function of obtaining distance information. For example, at least one of the imaging sectionstomay be a stereo camera constituted of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

12051 12111 12114 12100 12101 12104 12100 12100 12051 For example, the microcomputercan determine a distance to each three-dimensional object within the imaging rangestoand a temporal change in the distance (relative speed with respect to the vehicle) on the basis of the distance information obtained from the imaging sectionsto, and thereby extract, as a preceding vehicle, a nearest three-dimensional object in particular that is present on a traveling path of the vehicleand which travels in substantially the same direction as the vehicleat a predetermined speed (for example, equal to or more than 0 km/hour). Further, the microcomputercan set a following distance to be maintained in front of a preceding vehicle in advance, and perform automatic brake control (including following stop control), automatic acceleration control (including following start control), or the like. It is thus possible to perform cooperative control intended for automated driving that makes the vehicle travel automatedly without depending on the operation of the driver or the like.

12051 12101 12104 12051 12100 12100 12100 12051 12051 12061 12062 12010 12051 For example, the microcomputercan classify three-dimensional object data on three-dimensional objects into three-dimensional object data of a two-wheeled vehicle, a standard-sized vehicle, a large-sized vehicle, a pedestrian, a utility pole, and other three-dimensional objects on the basis of the distance information obtained from the imaging sectionsto, extract the classified three-dimensional object data, and use the extracted three-dimensional object data for automatic avoidance of an obstacle. For example, the microcomputeridentifies obstacles around the vehicleas obstacles that the driver of the vehiclecan recognize visually and obstacles that are difficult for the driver of the vehicleto recognize visually. Then, the microcomputerdetermines a collision risk indicating a risk of collision with each obstacle. In a situation in which the collision risk is equal to or higher than a set value and there is thus a possibility of collision, the microcomputeroutputs a warning to the driver via the audio speakeror the display section, and performs forced deceleration or avoidance steering via the driving system control unit. The microcomputercan thereby assist in driving to avoid collision.

12101 12104 12051 12101 12104 12101 12104 12051 12101 12104 12052 12062 12052 12062 At least one of the imaging sectionstomay be an infrared camera that detects infrared rays. The microcomputercan, for example, recognize a pedestrian by determining whether or not there is a pedestrian in imaged images of the imaging sectionsto. Such recognition of a pedestrian is, for example, performed by a procedure of extracting characteristic points in the imaged images of the imaging sectionstoas infrared cameras and a procedure of determining whether or not it is the pedestrian by performing pattern matching processing on a series of characteristic points representing the contour of the object. When the microcomputerdetermines that there is a pedestrian in the imaged images of the imaging sectionsto, and thus recognizes the pedestrian, the sound/image output sectioncontrols the display sectionso that a square contour line for emphasis is displayed so as to be superimposed on the recognized pedestrian. The sound/image output sectionmay also control the display sectionso that an icon or the like representing the pedestrian is displayed at a desired position.

12031 12030 101 101 101 101 101 101 101 12031 12030 12031 12030 7 FIG. 11 FIG. 19 FIG. 23 FIG. 24 FIG. 25 FIG. 26 FIG. An example of the vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to a configuration obtained by combining the imaging sectionand the outside-vehicle information detecting unitamong the configurations described above. Specifically, the signal processing deviceof, the signal processing deviceA of, the signal processing deviceB of, the signal processing deviceC of, the signal processing deviceD of, the signal processing deviceE of, and the signal processing deviceF ofcan be applied to a configuration obtained by combining the imaging sectionand the outside-vehicle information detecting unit. By applying the technology according to the present disclosure to a configuration obtained by combining the imaging sectionand the outside-vehicle information detecting unit, it is possible to reduce the effects of latency that may occur in the processing of detecting an object or the processing of detecting the distance thereto.

Note that, in the present specification, a system refers to a collection of a plurality of constituent elements (devices, modules (parts), and the like), and it does not matter whether all the constituent elements are in the same housing. Accordingly, both a plurality of devices accommodated in separate housings and connected together via a network and a single device in which a plurality of modules are accommodated in one housing are systems.

Further, embodiments of the present disclosure are not limited to the embodiments described above, and various changes can be made thereto without departing from the spirit and scope of the present disclosure.

For example, the present disclosure may take a form of cloud computing in which one function is shared and performed together by a plurality of devices via a network.

Further, the steps described with reference to the flowcharts described above may be executed by a single device or may be executed by a plurality of devices in a shared manner.

Furthermore, in a case in which a single step includes a plurality of processes, the plurality of processes included in the single step may be executed by a single device or may be executed by a plurality of devices in a shared manner.

(1) A light detection device including: an event detecting section configured to output event data representing occurrence of an event that is a change in an electric signal, the event being detected in an event detection pixel configured to generate the electric signal by performing photoelectric conversion, and a plurality of the event detection pixels being disposed on a two-dimensional plane; a received light amount calculation section configured to calculate a received light amount of the event detection pixel; and a latency correction section configured to correct information related to latency in the event data on the basis of the received light amount of the event detection pixel calculated by the received light amount calculation section. (2) The light detection device according to (1), wherein the latency correction section includes a table registering a relationship between the received light amount and the latency, reads out, from the table, the latency corresponding to the received light amount calculated by the received light amount calculation section, and corrects the event data by a latency correction amount consisting of the readout latency. (3) The light detection device according to (2), wherein the event data includes time information indicating when the event is detected by the event detecting section, and the latency correction section corrects the time information about the event data by the latency correction amount corresponding to the received light amount. (4) The light detection device according to (3), wherein the event data further includes coordinate information indicating a pixel position of each of the plurality of event detection pixels of the event detecting section, and the latency correction section corrects, in association with the coordinate information about each of the plurality of event detection pixels, the time information about the event data by the latency correction amount corresponding to the received light amount. (5) The light detection device according to (3), wherein the event data further includes coordinate information indicating a pixel position of each of the plurality of event detection pixels of the event detecting section, and the latency correction section corrects the event data by adding corrected time information obtained by correcting the time information by the latency correction amount corresponding to the received light amount or by adding the latency correction amount, in association with the coordinate information about each of the plurality of event detection pixels. (6) The light detection device according to (3), wherein the event data further includes information about a type of the event indicating whether the event is a positive event in which the electric signal detected by the event detecting section changes in a positive direction or a negative event in which the electric signal changes in a negative direction, the table registers a relationship between the received light amount and the latency for each information item about the type of the event, and the latency correction section corrects the event data by adding corrected time information obtained by correcting the time information by the latency correction amount corresponding to the received light amount or by adding the latency correction amount, in accordance with the type of the event and in association with the coordinate position of each of the plurality of event detection pixels. (7) The light detection device according to (1), further including: a visible light detecting section configured to detect the received light amount of visible light in a vicinity of an imaging surface configured by disposing the plurality of event detection pixels on a two-dimensional plane and output an output signal corresponding to the received light amount that is detected, wherein the received light amount calculation section calculates the received light amount of the event detection pixel on the basis of the output signal. (8) The light detection device according to (7), wherein the visible light detecting section detects the received light amount of the visible light at one point in the vicinity of the imaging surface configured by disposing the plurality of event detection pixels on the two-dimensional plane and outputs the received light amount of the visible light as the corresponding output signal, the received light amount calculation section calculates the received light amount of each of all the event detection pixels as the same received light amount on the basis of the output signal, and the latency correction section corrects the event data of each of all the event detection pixels by a latency correction amount corresponding to the same received light amount. (9) The light detection device according to (1), further including: a gradation signal detecting section including, in a vicinity of an imaging surface configured by disposing the plurality of event detection pixels on the two-dimensional plane, a plurality of imaging pixels disposed on a two-dimensional plane and configured to detect and output a gradation signal corresponding to a light amount of incident light, wherein the received light amount calculation section calculates the received light amount of each of the plurality of event detection pixels on the basis of the gradation signal of each of the plurality of imaging pixels. (10) The light detection device according to (9), wherein the received light amount calculation section calculates the received light amount for each of the plurality of imaging pixels, and applies the calculated received light amount as the received light amount of the event detection pixel at a position corresponding to a position of the imaging pixel, and the latency correction section corrects the event data of each of the plurality of event detection pixels by a latency correction amount corresponding to the received light amount applied to each of the plurality of event detection pixels. (11) The light detection device according to (10), wherein the received light amount calculation section calculates the received light amount of the event detection pixel on the basis of a predetermined spectrum and the gradation signal. (12) The light detection device according to (11), wherein the predetermined spectrum is a spectrum estimated from the gradation signal. (13) The light detection device according to (12), wherein the plurality of imaging pixels constituting the gradation signal detecting section are provided with a plurality of filters configured to perform filtering such that the transmitted incident light is divided into a plurality of wavelength bands, the plurality of filters being provided in a predetermined array at preceding stages of the plurality of imaging pixels and for each of the plurality of imaging pixels, and the received light amount calculation section estimates the spectrum from a ratio of the received light amount based on the gradation signal for each of the plurality of filters, and calculates the received light amount of the event detection pixel on the basis of the estimated spectrum and the gradation signal. (14) The light detection device according to (9), wherein the received light amount calculation section calculates the received light amount for each of the plurality of imaging pixels and applies the calculated received light amount as the received light amount of the event detection pixel at a position corresponding to a position of the imaging pixel, and the latency correction section corrects the event data of each of the plurality of event detection pixels by adding the received light amount applied to each of the plurality of event detection pixels to the event data. (15) The light detection device according to (9), wherein the plurality of event detection pixels and the plurality of imaging pixels are disposed on the same two-dimensional plane, and the event detecting section and the gradation signal detecting section are integrated. (16) The light detection device according to (15), wherein the received light amount calculation section is integrated in addition to the event detecting section and the gradation signal detecting section. (17) The light detection device according to (16), wherein the latency correction section is integrated in addition to the event detecting section, the gradation signal detecting section, and the received light amount calculation section. (18) The light detection device according to (1), wherein the latency correction section corrects the information related to the latency of the event data by shifting, on the basis of the received light amount of each of the plurality of event detection pixels calculated by the received light amount calculation section, an event data group consisting of a predetermined quantity of the time-series event data to be output to a subsequent stage, by an event data quantity corresponding to the latency, and outputting the shifted event data. (19) A light detection method including: outputting event data representing occurrence of an event that is a change in an electric signal, the event being detected in an event detection pixel configured to generate the electric signal by performing photoelectric conversion, a plurality of the event detection pixels being disposed on a two-dimensional plane; calculating a received light amount of the event detection pixel; and correcting information related to latency in the event data on the basis of the calculated received light amount of the event detection pixel. (20) A program that causes a computer to function as: an event detecting section configured to output event data representing occurrence of an event that is a change in an electric signal, the event being detected in an event detection pixel configured to generate the electric signal by performing photoelectric conversion, a plurality of the event detection pixels being disposed on a two-dimensional plane; a received light amount calculation section configured to calculate a received light amount of the event detection pixel; and a latency correction section configured to correct information related to latency in the event data on the basis of the received light amount of the event detection pixel calculated by the received light amount calculation section. Note that the present disclosure can also adopt configurations such as the following.

101 101 101 111 112 113 113 113 114 114 121 121 121 122 122 122 141 141 1 141 2 141 151 171 201 201 201 221 221 221 222 222 222 223 223 223 224 224 224 241 241 261 270 271 272 ,A toF Signal processing device,Event signal detection sensor,Visible light sensor,,A,B Subsequent-stage ISP/AP,,B Subsequent-stage application,,A,B Event detection pixel received light amount calculation section,,A,B Latency correction section,,A-,A-,B Lens,CMOS image sensor,Latency correction section,,D,E Hybrid EVS sensor,,D,E Event data acquisition section,,D,E Gradation signal acquisition section,,D,E Event detection pixel received light amount calculation section,,D,E Latency correction section,,E Subsequent-stage ISP/AP,Switching sensor,Pixel array,Event data detecting section,Gradation signal acquisition section