Distance Measuring Apparatus, Distance Measuring Method, and Storage Medium

The present disclosure relates to a distance measuring apparatus, a distance measuring method, a storage medium, and the like.

As an apparatus that performs distance measurement to an object, a distance measuring camera system using the triangulation principle is mounted in many automobiles for the purpose of realization of Advanced Driver Assistance System (ADAS) functionality. Examples of the distance measuring camera system include a stereo camera system and an image capturing surface phase difference distance measuring system.

The stereo camera system arranges two cameras separated by a predetermined interval (baseline length) in parallel, detects an amount of deviation caused by parallax of the object captured in images captured using each of the cameras, and calculates a distance to the object based on the amount of deviation.

The image capturing surface phase difference distance measuring system is a system configured to perform image capturing and perform distance measurement using a single camera provided with an image capturing element called an image capturing surface phase difference image capturing element. That is, two images having a phase difference are generated by photoelectrically converting light that has passed through different pupils of an imaging optical system using a plurality of photoelectric conversion units on the image capturing surface phase difference image capturing element, and a distance to the object is calculated based on the above-described phase difference.

These distance measuring camera systems may incorrectly calculate the distance to the object in bad weather conditions, for example. Bad weather conditions refer to, for example, situations in which particles of rain, fog, snow, and the like hinder visibility of the object.

In the above-described distance measuring camera system, if light scattered by particles located in front of the object is captured by the camera, image signals corresponding to objects other than the target object are generated. As a result, there are cases in which the distance to the object cannot be accurately calculated due to the influence of the image signals.

In contrast, Japanese Patent No. 6293134 discloses a configuration in which each camera acquires images clearly capturing only subjects in a certain distance range by performing range gate control on a light source and a stereo camera. Range gate control emits pulsed light at a predetermined cycle in front of the camera, and the image capturing element inside the camera performs an exposure operation (accumulation operation) at a predetermined timing according to the target distance. Thereby, clear image capturing of only the object at the target distance can be performed.

Japanese Patent No. 6293134 further discloses that, by thus performing range gate control on two cameras as described above, an amount of deviation caused by parallax of the object captured in each acquired range gate image is detected, and the distance to the object is calculated based on this. By using this technology, the distance to the object can be accurately calculated even in bad weather conditions.

However, in the technology of Japanese Patent No. 6293134 described above, synchronization of images acquired using two cameras on the order of nanoseconds is necessary in order to measure distance to the object with high accuracy under bad weather conditions. For example, in order to acquire a range gate image of a target distance range of 3 m, assuming the speed of light is 300,000 km/s, the required exposure time of the camera becomes 20 ns.

Accordingly, two range gate images synchronized on the order of ns become necessary in order to generate a stereo distance measuring image having high accuracy. However, synchronization of two cameras on the order of ns requires wiring length, accuracy of clock phase, and the like, and there is an issue wherein practical application is difficult.

The present disclosure relates to a distance measuring apparatus comprising: a light emitter configured to emit pulsed light in a traveling direction of a movable apparatus, a photoelectric conversion element having a plurality of pixels arranged, each pixel having a first photoelectric conversion unit configured to generate a first photoelectric conversion signal and a second photoelectric conversion unit configured to generate a second photoelectric conversion signal, wherein the first photoelectric conversion signal and the second photoelectric conversion signal have a predetermined parallax, a visibility condition determination unit configured to determine a visibility condition of the traveling direction, and a control unit configured to switch, based on the visibility condition, whether to execute processing for calculating a distance to an object based on the first photoelectric conversion signal and the second photoelectric conversion signal generated without performing the range gate control, or to execute processing for calculating a distance to an object based on the first photoelectric conversion signal and the second photoelectric conversion signal generated by the range gate control.

Further features of the present disclosure will become apparent from the following description of embodiments with reference to the attached drawings.

Hereinafter, with reference to the accompanying drawings, favorable modes of the present disclosure will be described using Embodiments. In each diagram, the same reference signs are applied to the same members or elements, and duplicate description will be omitted or simplified.

100 300 200 100 300 200 Hereinafter, an explanation is provided with respect to the First Embodiment of the present disclosure. In the First Embodiment, although a cameraand a light emitterare attached to a movable apparatussuch as a vehicle, the attachment configuration is not limited thereto. It should be noted that details of the cameraand the light emitter, and the movable apparatusare described below.

1 FIG.A 1 FIG.D 1 FIG.A 1 FIG.B 100 200 100 200 200 toare diagrams showing examples of attachment positions of the camerato the movable apparatusaccording to the First Embodiment. The cameramay be attached to an interior of the movable apparatus, for example, an interior side of a front window of the movable apparatusas shown in, or on a dashboard as shown in.

100 200 200 1 FIG.C 1 FIG.D In addition, the cameramay be attached to an exterior of the movable apparatus, for example, a headlamp peripheral portion of the movable apparatusas shown in, or for example, a side mirror peripheral portion as shown in.

2 FIG.A 2 FIG.C 2 FIG.A 2 FIG.B 2 FIG.C 300 200 300 200 toare diagrams showing examples of attachment positions of the light emitteron the movable apparatusaccording to the First Embodiment. The light emittermay be attached to a headlamp peripheral portion of the movable apparatusas shown in, a roof portion as shown in, or a side mirror peripheral portion as shown in.

3 FIG. 3 FIG. 100 300 200 100 300 200 is a functional block diagram showing a configuration example of the camera, the light emitter, and the movable apparatusaccording to the First Embodiment. It should be noted that a part of the functional blocks shown inis realized by executing a computer program stored in a memory serving as a storage medium (not shown) in a computer (not shown) included in each of the camera, the light emitter, and the movable apparatus.

3 FIG. However, a part or all of the functional blocks may be implemented by hardware. Hardware, dedicated circuits (ASIC), processors (reconfigurable processor, DSP), and the like may be used. In addition, each of the functional blocks shown inneed not be incorporated in the same housing, and may be configured by separate apparatuses connected to each other via signal paths.

100 101 102 103 104 105 106 107 108 The cameraincludes an imaging optical system, a photoelectric conversion element, an image processing unit, a distance measuring unit, a camera control unit, a storage unit, a communication unit, a bad weather determination unit, and the like.

101 102 102 The imaging optical systemforms an image (optical image) of an object on the photoelectric conversion element, and has an exit pupil at a position separated from the photoelectric conversion elementby a predetermined distance.

102 102 The photoelectric conversion elementis a semiconductor image sensor element such as a CMOS (Complementary Metal Oxide Semiconductor) sensor and the like. The photoelectric conversion elementis provided with a pixel region in which a plurality of pixels having a photoelectric conversion function are arranged in two dimensions, and each pixel region has a microlens and two photoelectric conversion units.

102 An object image formed on the photoelectric conversion elementvia the microlens is photoelectrically converted by the two photoelectric conversion units to generate a first photoelectric conversion signal and a second photoelectric conversion signal, respectively. It should be noted that one of the two photoelectric conversion units receives light from one exit pupil of the imaging optical system, and the other photoelectric conversion unit receives light from a different exit pupil of the imaging optical system.

103 102 103 104 105 201 200 The image processing unitperforms image processing such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, data compression, and the like on an image signal output from the photoelectric conversion element, and generates a final image signal. The output of the image processing unitis supplied to the distance measuring unitand the camera control unit, and is also supplied to an ECU(Electric Control Unit) of the movable apparatus.

104 103 104 103 The distance measuring unitperforms recognition processing of objects such as surrounding people, vehicles, and the like by performing image recognition based on the image signal supplied from the image processing unit. In addition, the distance measuring unitcalculates a distance to the object based on a phase difference (deviation due to parallax) of the first photoelectric conversion signal and the second photoelectric conversion signal supplied from the image processing unit.

105 105 100 The camera control unitincorporates a CPU serving as a computer, and a memory storing a computer program, and the camera control unitperforms control of each unit of the cameraby the CPU executing the computer program stored in the memory.

105 102 105 300 107 It should be noted that the camera control unitfunctions as a control unit, and performs timing control of the length of an exposure period (charge accumulation period) of each frame and various control signals, and the like by transmitting a reference clock signal that is repeatedly output at predetermined intervals to the photoelectric conversion elementand the like. In addition, the camera control unittransmits the reference clock signal and control signals to the light emittervia the communication unit.

106 The storage unitincludes a recording medium such as a memory card, a hard disk, and the like, and can store and read out image signals.

107 100 108 200 105 108 The communication unitincludes a wireless or wired interface, outputs image signals generated to the outside of the camera, and receives various signals from the outside. The bad weather determination unitdetermines a visibility condition according to weather conditions and the like around the front of the movable apparatus, and outputs a determination result to the camera control unit. That is, the bad weather determination unitfunctions as a visibility condition determination unit that determines the visibility condition (such as fog, rain, snow, and the like) in the traveling direction of the movable apparatus.

200 200 100 200 Weather conditions may be acquired based on weather information obtained from the web and position information of the movable apparatusbased on GPS. Alternatively, technologies described in Japanese Patent Application Laid-Open No. 2008-33872 may be used. That is, the bad weather condition may be acquired based on a brightness difference between a range directly illuminated by headlights of the movable apparatusand a range not directly illuminated, captured by the cameraattached to the movable apparatus.

300 301 302 303 303 107 100 105 302 302 The light emitterincludes a light emitting unit, a light emission control unit, and a communication unit. The communication unitcommunicates with the communication unitof the camera, receives setting information, control signals, and reference clock signals from the camera control unitto the light emission control unit, and supplies the setting information, control signals, and reference clock signals to the light emission control unit.

302 105 100 303 301 The light emission control unitreceives the reference clock signal transmitted by the camera control unitof the cameravia the communication unit, generates a pulse signal at a predetermined timing based on the reference clock signal, and outputs the pulse signal to the light emitting unit.

302 Here, the light emission control unitis capable of setting a period from the reference clock signal until outputting a pulse, a pulse output width, a pulse non-output width, and a repetition cycle from a pulse output to a next pulse output, a number of repetitions, and the like.

105 302 107 303 301 300 302 100 When the camera control unittransmits a predetermined control signal to the light emission control unitvia the communication unitand the communication unit, a pulse signal is output to the light emitting unitat a predetermined timing based on the reference clock signal, and a light emission period of the light emitteris controlled. In this manner, the light emission control unitis emission-controlled in synchronization with the reference clock signal input to the camera.

301 200 300 105 The light emitting unitis, for example, a near-infrared LED disposed at the front of the movable apparatus, and is configured by a lens and a light emitting unit. That is, the light emittercan emit pulsed light in the traveling direction of the movable apparatus. It should be noted that, in the present embodiment, the camera control unitfunctions as a control unit that performs range gate control of a light emission time of the light emitter, an exposure time of the photoelectric conversion element, and an exposure timing. In the following embodiments, the exposure time and the exposure timing respectively mean a photoelectric conversion time and a photoelectric conversion timing.

301 301 200 301 302 It should be noted that the frequency band of light irradiated by the light emitting unitis not limited thereto, and may be, for example, in the visible light region. That is, the light emitted by the light emitter is visible light or non-visible light. In addition, the light emitting unitmay be a light emitter provided on the movable apparatus, such as a headlight. The light emitting unitoutputs pulsed light for a predetermined light emission time in accordance with the pulse signal output from the light emission control unit.

102 300 300 102 300 In this manner, the same reference clock signal supplied to the photoelectric conversion elementis also transmitted to the light emitter, and the light emitterexecutes light emission control based on the reference clock signal. Thereby, an exposure (charge accumulation) timing of the photoelectric conversion elementand light emission timing by the light emitterare synchronized.

201 200 201 200 202 The ECUof the movable apparatusincorporates a CPU serving as a computer and a memory storing a computer program. In addition, the ECUperforms control of each unit in the movable apparatus, such as a vehicle control unitand the like, by the CPU executing the computer program stored in the memory.

104 202 203 201 202 201 The output of the distance measuring unitis supplied to the vehicle control unitand a display unitvia the ECU. The vehicle control unitfunctions as a movement control unit that performs driving, stopping, direction control, and the like of the vehicle serving as the movable apparatus based on the output of the ECU.

203 200 In addition, the display unitfunctions as a display means, includes display elements such as a liquid crystal device, an organic EL, and the like, and is mounted on, for example, a dashboard or the like of the movable apparatus.

201 104 201 104 203 In the present embodiment, the ECUreceives information of distance measurement results from the distance measuring unit, and is capable of executing vehicle stop control (such as automatic braking and the like) according to the content of the distance measurement results. In addition, the ECUreceives distance measurement processing data from the distance measuring unit, and transmits the distance measurement processing data to the display unit.

201 203 200 102 104 Based on the output of the ECU, the display unitdisplays, by using, for example, a GUI, various information to a driver of the movable apparatus, including an image generated by the photoelectric conversion element, a distance measurement result by the distance measuring unit, and a traveling state of the vehicle, and the like.

103 104 200 200 200 3 FIG. It should be noted that the image processing unit, the distance measuring unit, and the like inneed not be mounted on the movable apparatus, and may be provided in an external terminal provided separately from the movable apparatusfor remotely controlling the movable apparatus, or monitoring travel of the movable apparatus.

102 4 FIG. 5 FIG. Next, an explanation is provided with respect to a distance measuring principle using the image capturing surface phase difference image capturing element utilizing the photoelectric conversion elementby usingand.

4 FIG.A 4 FIG.B 4 FIG.A 102 102 102 400 400 andare schematic diagrams showing a configuration example of the photoelectric conversion elementaccording to the First Embodiment, andis a plan view of the photoelectric conversion elementviewed from a light incident direction. The photoelectric conversion elementis configured by arranging a plurality of 2-row×2-column pixel groupsin a matrix. The pixel grouphas four infrared pixels IR that detect IR (infrared) light.

400 300 400 It should be noted that the arrangement of each pixel in the pixel groupis not limited thereto, and may be changed according to a wavelength band of light emitted from the light emitter. The pixel groupmay be configured by, for example, an infrared pixel IR that detects IR light, a red pixel that detects red color, a green pixel that detects green light, and a blue pixel that detects blue light.

400 In addition, the pixel groupmay have two green pixels that detect green color, a red pixel that detects red color, and a blue pixel that detects blue light. In a case in which the above-described pixel arrangement is used, the two green pixels are arranged diagonally.

4 FIG.B 4 FIG.A 400 414 415 414 413 415 is a cross-sectional view taken along line I-I of the pixel groupin. Each pixel has a light guide layer, and a light receiving layer. The light guide layerhas the microlensfor efficiently guiding light incident on the pixel to the light receiving layer, a color filter that transmits light of a wavelength band detected by each pixel, and wiring for pixel readout and pixel driving.

415 414 415 411 412 411 412 The light receiving layeris a photoelectric conversion unit that outputs light entered via the light guide layeras an electrical signal by photoelectric conversion. The light receiving layerhas a first photoelectric conversion unitand a second photoelectric conversion unit. In this manner, each pixel has the first photoelectric conversion unitand the second photoelectric conversion unit.

5 FIG.A 5 FIG.D 5 FIG.A 500 101 102 102 toare schematic diagrams for explaining a relationship between a distance of an object and incident light in the image capturing surface phase difference method.is a schematic diagram showing an exit pupilof the imaging optical system, and light incident on each light source conversion unit of an infrared pixel IR of the photoelectric conversion element. Although the photoelectric conversion elementhas a plurality of pixels, for simplification, an explanation is provided with respect to one infrared pixel IR.

413 500 415 510 500 411 520 412 The microlensof the infrared pixel IR is arranged so that the exit pupiland the light receiving layerare in an optically conjugate relationship. As a result, light that has passed through a first pupil regionthat is a partial pupil region included in the exit pupilis incident on the first photoelectric conversion unit. Similarly, light that has passed through a second pupil regionthat is a partial pupil region is incident on the second photoelectric conversion unit.

411 411 102 102 510 The first photoelectric conversion unitof each pixel outputs a first photoelectric conversion signal by photoelectrically converting received light. A first image signal is generated from the first photoelectric conversion signals output from a plurality of first photoelectric conversion unitsincluded in the photoelectric conversion element. The first image signal indicates an intensity distribution of an image formed on the photoelectric conversion elementby light that has mainly passed through the first pupil region.

412 412 102 102 520 The second photoelectric conversion unitof each pixel outputs a second photoelectric conversion signal by photoelectrically converting received light. A second image signal is generated from the second photoelectric conversion signals output from a plurality of second photoelectric conversion unitsincluded in the photoelectric conversion element. The second image signal indicates an intensity distribution of an image formed on the photoelectric conversion elementby light that has mainly passed through the second pupil region.

In this manner, in the photoelectric conversion element of the present embodiment, a plurality of pixels are arranged, each having a first photoelectric conversion unit that generates a first photoelectric conversion signal and a second photoelectric conversion unit that generates a second photoelectric conversion signal, wherein the first photoelectric conversion signal and the second photoelectric conversion signal have a predetermined parallax.

5 FIG.B 5 FIG.D A relative amount of positional deviation between the first image signal and the second image signal (hereinafter, this is referred to as a parallax amount or a phase difference) corresponds to the defocus amount. An explanation is provided with respect to a relationship between the parallax amount and the defocus amount by usingto.

5 FIG.B 5 FIG.D 102 101 511 510 521 520 toare schematic diagrams showing a relationship between the photoelectric conversion element, the imaging optical system, and an object position. In the figures, reference numeralindicates first light that passes through the first pupil region, and reference numeralindicates second light that passes through the second pupil region.

5 FIG.B 511 521 102 511 521 shows a state at the time of focusing, and the first lightand the second lightconverge on the photoelectric conversion unit. At this time, a parallax amount between the first image signal formed by the first lightand the second image signal formed by the second lightbecomes 0.

5 FIG.C 511 521 shows a state defocused in a negative direction of the w axis on the image side. At this time, the parallax amount between the first image signal formed by the first lightand the second image signal formed by the second lightdoes not become 0, and has a negative value.

5 FIG.D 511 521 shows a state that is defocused in the positive direction of the w axis on the image side. At this time, the parallax amount between the first image signal formed by the first lightand the second image signal formed by the second lightdoes not become 0, and has a positive value.

5 FIG.C 5 FIG.D From the comparison betweenand, it can be seen that the direction in which parallax occurs is reversed according to the positive or negative of the defocus amount. In addition, from the geometric relationship, it can be understood that a parallax amount according to the defocus amount occurs.

Accordingly, the parallax amount between the first image signal and the second image signal can be detected by a region-based matching method, for example, a block matching method, and converted to a defocus amount. Here, the block matching method is a method in which a region having a high degree of similarity is searched from another image for a certain region selected from one image, and a positional deviation of the region having the high degree of similarity is taken as a parallax.

101 101 101 102 Furthermore, by using an imaging formula of the imaging optical system, the defocus amount on the image side can be converted to a distance to an object. The imaging formula of the imaging optical systemis represented by the following Equation (1) when a focal length of the imaging optical systemis f, a distance from an image-side principal point to the photoelectric conversion elementis Ipp, a defocus amount is ΔL, and a distance to the object is D.

104 The above is an explanation of the distance measuring principle using the image capturing surface phase difference method. The distance measuring unitof the present embodiment calculates a distance to the object by operations such as described above based on the first photoelectric conversion signal and the second photoelectric conversion signal.

6 FIG. 300 100 is a diagram showing progress of light radiated from the light emitterand reflected light thereof, and exposure timing of the camerain range gate control according to the First Embodiment.

6 FIG. By using, an explanation is provided with respect to a method of acquiring an image (range gate image) of a target distance by performing control (range gate control) that synchronizes light emission timing and exposure timing according to the target distance.

6 FIG. It should be noted that a camera that acquires an image of the target distance by range gate control in this manner is called a range gate camera. In, the horizontal axis shows distance, and the vertical axis shows time.

610 1 2 620 3 6 FIG. First, an explanation is provided with respect to the horizontal axis. Fogexists between distance xand distance xdue to bad weather, and a vehicleexists at distance x. In addition, in, in range gate control, a position of distance D serves as a starting point, and from there, a range gate image in a range of target distance range R is acquired.

620 In this case, the target distance range R becomes a target distance range for which image capturing is desired to be performed. At this time, the vehicleexists within the target distance range R.

0 300 Next, an explanation is provided with respect to the vertical axis. Timeis set as light emission start timing at the light emitter, and time tf is set as light emission end timing. At this time, the light emission period becomes tf.

1 2 102 In addition, in a case in which a position of distance D serves as a starting point, and a range gate image in a range of target distance range R is acquired from there, exposure start time is set as time t, and exposure end time is set as time t. It should be noted that in the explanation of the present embodiment, “exposure” means photoelectric conversion in the photoelectric conversion element, “exposure start” means photoelectric conversion start, and “exposure end” means photoelectric conversion end.

1 300 0 100 2 300 100 It should be noted that time tis a timing at which radiated light that the light emitterradiated at timehas returned to the cameraas reflected light from distance D. In addition, tis a timing at which radiated light that the light emitterradiated has returned to the cameraas reflected light from a location that has advanced by target distance range R from distance D.

6 FIG. 100 3 610 100 4 In addition, in, a timing at which first reflected light returns to the camerais set as time t, and a timing at which last reflected light caused by the fogreturns to the camerais set as time t.

3 4 610 100 1 2 620 610 In range gate control, exposure is not performed during a period from time tto time tin which reflected light of the fogreaches the camera, and exposure (photoelectric conversion) is performed only during a period from time tto time tin which reflected light of the portion from distance D to target distance range R arrives. Accordingly, an image of the vehiclecan be clearly acquired while removing the fog.

100 0 Here, an explanation is provided with respect to time until reflected light from an object existing at distance x returns to the camera. A timing at which radiated light that started light emission at timehits an object existing at distance x and returns to the image capturing unit as reflected light is set as time tr. At this time, a relationship between timing time tr at which the reflected light returns and distance x to the image capturing object becomes the following Equation (2).

6 FIG. 1 As shown in, when distance D to target distance range R is set as the image capturing range, exposure timing time tof the starting point of the range can be obtained as the following Equation (3) by substituting distance D for distance x in the above Equation (2).

2 In addition, exposure timing time tof the end point of the range can be obtained as the following Equation (4) by substituting distance (D+R) in the above Equation (2) and adding time tf.

1 2 In this manner, according to distance x desired to be performed image capturing (target distance range R), time tf from light emission start to light emission end, time tfrom light emission start until exposure start, and time tuntil exposure end are controlled. Thereby, range gate control that can clearly perform image capturing of an object at the target distance even when fog and the like exist between the camera and the target distance is realized.

7 FIG. is a timing chart showing examples of light emission and exposure control operations in a one-frame period in range gate control according to the First Embodiment.

7 FIG. 300 300 102 In, “vertical synchronization signal” indicates a frame cycle of image capturing, and a period between a Low pulse and a next Low pulse is a one-frame period. “Light emission control” indicates light emission timing of the light emitter, and during a high level, light emission by the light emitteris performed. “Exposure control of the photoelectric conversion element” is an exposure control signal supplied from the camera control circuit, and during a high level, photoelectric conversion in the photoelectric conversion elementis performed.

That is, for example, at a rising edge of the exposure control signal, charge accumulation is started by turning off a reset switch (not shown) for resetting charge of the photoelectric conversion unit. Then, at the falling edge of the exposure control signal, charge accumulated in the photoelectric conversion unit is transferred to and held in a memory (not shown) in the pixel, and then the reset switch is turned on again.

7 FIG. 102 In range gate control, the light emission period is controlled in a pulse-like manner as in, and exposure (photoelectric conversion) of the photoelectric conversion elementis performed only with respect to reflected light of light from a specific target distance range R.

7 FIG. 1 2 1 100 In, as described above, tf indicates an emission period from emission start to emission end, tcorresponds to a time from emission start to exposure start, and tcorresponds to a time from emission start to exposure end. In addition, as described above, tindicates a period from emission start until light reaches a specific target distance range R, and reflected light returns to the camera.

1 2 In addition, as described above, the time from tto tbecomes a period during which reflected light of the specific target distance range R is exposed. In order to correctly perform range gate control, synchronizing timing of light emission start and exposure start according to a range of a predetermined target distance is necessary.

105 102 302 102 302 In addition, as described above, in the present embodiment, the camera control unitsynchronizes the photoelectric conversion elementand the light emission control unitby transmitting the same reference clock signal to the photoelectric conversion elementand the light emission control unit. A period from light emission start to next light emission start indicated by light emission control on the timing chart becomes a range gate operation cycle.

102 102 Then, light exposed by the photoelectric conversion elementin one range gate operation cycle is converted to charge and held in a memory in a pixel of the photoelectric conversion element.

7 FIG. 102 102 100 In that state, as shown in, a next range gate operation cycle is implemented within the same one-frame period, and light newly exposed by the photoelectric conversion elementis converted to charge and accumulatively stored in the memory in the pixel of the photoelectric conversion element. It should be noted that a period from light emission to the next light emission is set based on a time until the reflected light sufficiently attenuates and no longer returns to the camera.

7 FIG. 102 As shown in, the range gate operation cycle is implemented a predetermined plurality of times set within the one-frame period, and charge photoelectrically converted in the last range gate operation cycle within the one-frame period is also further accumulatively stored in the memory in the pixel of the photoelectric conversion element.

Thereafter, charge in the memory is read out via a vertical signal line, and thereafter, the charge of the memory is reset by a predetermined switch for reset.

300 100 300 In this manner, in the present embodiment, since the exposure period is synchronized with light emission by the light emitter, a clear image can be obtained for a targeted range even under bad weather conditions such as fog. In contrast, range gate control can accurately measure distance only for a target distance range R that depends on an exposure time of the cameraand a light emission time of the light emitter.

100 300 100 300 That is, it is necessary to reduce the target distance range in order to improve distance measurement accuracy. Accordingly, reducing the exposure time of the cameraand the light emission time of the light emitteris contemplated, although this is difficult due to physical control limitations of the cameraand the light emitter.

100 In addition, if range gate image capturing is attempted to a distance after reducing the target distance range R, there is an issue wherein power consumption becomes large because an exposure count of the cameraincreases. In addition, there is also an issue wherein measuring distance to a far distance takes time.

8 FIG. 8 FIG. To solve this problem, distance measurement processing in the distance measuring method according to the present embodiment will be described with reference to.is a flowchart showing an example of image generation processing in a distance measuring method according to the First Embodiment.

8 FIG. 105 It should be noted that operations of each step of the flowchart ofare sequentially performed by a CPU and the like serving as a computer in the camera control unitexecuting a computer program stored in a memory.

101 108 200 105 In step S, the bad weather determination unitacquires weather information regarding the area in front of the movable apparatusduring traveling by a method such as described above, and outputs the weather information to the camera control unit.

102 105 101 102 103 In step S, the camera control unitdetermines whether bad weather exists based on the weather information acquired in step S. In a case in which it is determined that bad weather does not exist in step S, the process proceeds to step S.

103 105 5 FIG.A In step S, the camera control unitstarts the image capturing surface phase difference distance measuring mode, thereby executing the image capturing surface phase difference distance measurement described in. Here, the image capturing surface phase difference distance measuring mode refers to a mode that generates a first photoelectric conversion signal and a second photoelectric conversion signal during an exposure time set according to brightness of the object.

411 102 412 102 101 That is, during only a predetermined exposure period within a one-frame period, the first photoelectric conversion signal is generated by the first photoelectric conversion unitof each pixel of the photoelectric conversion element, and the second photoelectric conversion signal is generated by the second photoelectric conversion unitof each pixel of the photoelectric conversion element, based on light that has passed through the imaging optical system.

103 It should be noted that the exposure time in step Sis set according to a luminance signal level of an image up to a previous frame period, for example. The luminance signal level of the image corresponds to brightness of the object.

104 103 Subsequently, in step S, image signals are acquired. That is, a first image signal and a second image signal are acquired based on the first photoelectric conversion signal and the second photoelectric conversion signal of a plurality of pixels. Then, each image signal is output to the image processing unit.

102 105 105 105 300 102 In contrast, in a case in which bad weather is determined in step S, the process proceeds to step S. In step S, an image capturing surface phase difference range gate distance measuring mode is started. That is, the camera control unitperforms range gate control of the light emitterand the photoelectric conversion element. Then, by range gate control, a first photoelectric conversion signal and a second photoelectric conversion signal for a target distance range are generated.

Here, the image capturing surface phase difference range gate distance measuring mode means generating a first photoelectric conversion signal and a second photoelectric conversion signal for a target distance range by range gate control.

106 105 103 In step S, the camera control unitacquires a first range gate image signal and a second range gate image signal based on the first photoelectric conversion signal and the second photoelectric conversion signal of a plurality of pixels for a specific target distance obtained by range gate control. Then, each range gate image signal is output to the image processing unit.

9 FIG.A 9 FIG.B andare timing charts showing examples of light emission and exposure control operations in a one-frame period in an image capturing surface phase difference range gate distance measuring mode according to the First Embodiment.

9 FIG.A 100 1 200 is a diagram showing an example of light emission timing and exposure timing per one-frame period in a case in which a position of the camerais set as a reference of 0 m, and a target distance rangeis set as, for example, 0 m to 30 m. It should be noted that the target distance may be changed according to a movement speed of the movable apparatus, for example, to become farther as the movement speed becomes faster.

1 300 411 412 For example, the target distance rangeis set to 0 m to 30 m. In that case, a light emission time from light emission start to light emission end of the light emitterbecomes 200 ns, a time from light emission start to exposure start at the first photoelectric conversion unitand the second photoelectric conversion unitbecomes 0 ns, and a time from light emission start to exposure end becomes 400 ns.

9 FIG.A 300 411 412 102 In the timing chart of, a period from light emission start of the light emitterto next light emission start becomes a range gate operation cycle. Light exposed (photoelectrically converted) by the first photoelectric conversion unitand the second photoelectric conversion unitin one range gate operation cycle is converted to charge and separately held in a memory (not shown) in the photoelectric conversion element.

411 412 102 In that state, a next range gate operation cycle is implemented, and light newly exposed by the first photoelectric conversion unitand the second photoelectric conversion unitis converted to charge, and separately accumulatively stored in the memory in the photoelectric conversion element.

411 412 In this manner, a first range gate image signal (A) and a second range gate image signal (A) in the target distance range 0 to 30 m are acquired at each of the first photoelectric conversion unitand the second photoelectric conversion unit.

1 411 1 412 Here, the first range gate image signal (A) refers to an image signal in the target distance range(0 m to 30 m) acquired by performing range gate control of the first photoelectric conversion unit. In addition, the second range gate image signal (A) is an image signal in the target distance range(0 m to 30 m) acquired by performing range gate control of the second photoelectric conversion unit.

9 FIG.B 100 2 is a diagram showing an example of light emission timing and exposure timing per one-frame period in a case in which the position of the camerais set as 0 m, and a target distance rangeis set as, for example, 30 m to 60 m.

2 300 411 412 The target distance rangeis set to 30 m to 60 m. In that case, the light emission time from light emission start to light emission end of the light emitterbecomes 200 ns, a time from light emission start to exposure start at the first photoelectric conversion unitand the second photoelectric conversion unitbecomes 200 ns, and a time from light emission start to exposure end becomes 600 ns.

9 FIG.A 9 FIG.A 300 2 411 412 Similar to, a period from light emission start of the light emitterto next light emission start in the timing chart becomes a range gate operation cycle. In addition, similar to, a first range gate image signal (B) and a second range gate image signal (B) in the target distance range(30 m to 60 m) are acquired at each of the first photoelectric conversion unitand the second photoelectric conversion unitin one range gate operation cycle.

2 411 2 412 The first range gate image signal (B) is an image signal in the target distance range(30 m to 60 m) acquired by performing range gate control of the first photoelectric conversion unit. In addition, the second range gate image signal (B) is an image signal in the target distance range(30 m to 60 m) acquired by performing range gate control of the second photoelectric conversion unit.

In the present embodiment, in this manner, the plurality of first photoelectric conversion signals generated by performing range gate control a plurality of times are added, and the plurality of second photoelectric conversion signals are added.

107 103 104 106 In step S, the image processing unitperforms image processing of each image signal acquired in step Sor step S. Here, image processing means generating a final image signal by black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaic processing, data compression, and the like.

108 104 107 108 5 FIG.A In step S, a parallax amount is calculated and converted to a distance value. In addition, an image is generated based on the distance value. That is, the distance measuring unitcalculates a parallax amount from each image signal acquired in step Sbased on a distance measuring principle by the image capturing surface phase difference method shown in, calculates a distance value, and generates a display image based on the distance value. It should be noted that in step S, image recognition is further performed based on each image signal.

108 104 It should be noted that in a case of the image capturing surface phase difference distance measuring mode, in step S, image recognition is performed based on the first image signal and the second image signal acquired in step S, and an object distance value is calculated based on a parallax amount of the first image signal and the second image signal.

108 106 In contrast, in a case of the image capturing surface phase difference range gate distance measuring mode, in step S, image recognition is performed based on the first range gate image signal (A) and the second range gate image signal (A) acquired in step S.

108 710 10 FIG.B In addition, in step S, a distance value is calculated based on a parallax amount of the first range gate image signal (A) and the second range gate image signal (A), and an imageas shown inis generated based on the distance value.

710 It should be noted that the imagemay be an image obtained by addition synthesis of both the first range gate image signal (A) and the second range gate image signal (A), or may be an image that displays only one of the first range gate image signal (A) and the second range gate image signal (A).

108 106 720 10 FIG.C In addition, in step S, a distance value is calculated based on a parallax amount of the first range gate image signal (B) and the second range gate image signal (B) acquired in step Sin a similar manner to the above, and an imageas shown inis generated based on the distance value.

720 It should be noted that the imagemay be an image obtained by addition synthesis of both the first range gate image signal (B) and the second range gate image signal (B), or may be an image that displays only one of the first range gate image signal (B) and the second range gate image signal (B).

710 720 730 10 FIG.D Thereafter, by performing image composition processing of the imageand the image, a composite imageof the 0 to 60 m distance range is generated as shown in.

710 720 Here, image composition is processing for obtaining an image of the 0 to 60 m distance range by superimposing the imagein the target distance range R 0 to 30 m and the imagein the target distance range R 30 m to 60 m.

10 FIG.A 10 FIG.D toare diagrams for explaining examples of composite images generated by the image capturing surface phase difference range gate distance measuring mode according to the First Embodiment.

10 FIG.A 10 FIG.A 700 103 700 shows one example of an imagethat is not range gate controlled in the present embodiment. That is,shows, for example, an example of an image generated in the image capturing surface phase difference distance measuring mode of step S. It should be noted that in a case in which the image is generated in the image capturing surface phase difference distance measuring mode, the imagemay be an image obtained by addition synthesis of both the first image signal and the second image signal, or may be an image that displays only one of the first image signal and the second image signal.

10 FIG.A 830 810 820 820 810 820 810 In, a pedestrian, fog, and a vehicleare captured. Although the vehicleexists beyond the fog, the vehicleis unclear due to the fogin the image that is not range gate controlled.

820 104 700 820 830 Accordingly, the vehiclecannot be accurately measured by distance measurement processing in the distance measuring unit. Although the imagethat is not range gate controlled can perform image capturing of the vehicleand the pedestrianin one frame, the image becomes unclear in bad weather conditions such as fog.

10 FIG.B 9 FIG.A 710 1 710 104 830 Next,shows the imagein the target distance range(0 m to 30 m) generated by the image capturing surface phase difference range gate distance measuring mode shown in. The imageis subjected to recognition processing by the distance measuring unit, and the pedestrianis detected.

411 412 105 710 1 710 In addition, by range gate control of the first photoelectric conversion unitand the second photoelectric conversion unitby the camera control unit, correspondence of the imageto the target distance range(0 m to 30 m) is understood, and those values (0 m, 30 m) are displayed on the image.

104 830 710 Furthermore, by image capturing surface phase difference distance measurement based on the first range gate image signal (A) and the second range gate image signal (A) by the distance measuring unit, the distance to the pedestrianis calculated as 5 m, and the calculated distance of 5 m is displayed on the image.

10 FIG.B 830 1 830 810 820 2 In, the pedestrianexisting in the target distance range(0 m to 30 m) is captured, and it is possible to accurately calculate the distance of the pedestrian. In contrast, the fogand the vehicleexisting in the target distance range(30 m to 60 m) are not captured.

10 FIG.C 9 FIG.B 720 2 Next,shows the imagein the target distance range(30 to 60 m) generated by the image capturing surface phase difference range gate distance measuring mode shown in.

720 810 820 104 In the image, because the fogis thinned by range gate control and the vehiclecan be captured as a clear image, the distance measuring unitcan accurately recognize and measure the distance.

720 104 820 411 412 105 820 2 720 The imageis subjected to recognition processing by the distance measuring unit, and the vehicleis detected. In addition, by range gate control of the first photoelectric conversion unitand the second photoelectric conversion unitby the camera control unit, it is understood that the vehiclecorresponds to the target distance range(30 m to 60 m), and those values (30 m, 60 m) are displayed on the image.

104 820 720 720 830 1 Furthermore, by image capturing surface phase difference distance measurement based on the first range gate image signal (B) and the second range gate image signal (B) by the distance measuring unit, the distance to the vehicleis calculated as 40 m, and is displayed on the image. In contrast, in the image, the pedestrianexisting in the target distance range(0 to 30 m) is not captured.

10 FIG.D 730 710 720 730 710 720 1 2 820 830 Next,shows the composite imagein 0 to 60 m obtained by compositing the imageand the image. In addition, the composite imagehas distance information of the imageand the image, and the distance range combined from the target distance rangeand the target distance range(0 m to 60 m) and the distance of the vehicleand the pedestrianare displayed.

200 730 820 810 820 201 201 202 Thereby, for example, a driver of the movable apparatuswho has visually recognized the composite imagecan understand that the vehicleexists beyond the fogand that the vehicleexists 40 m ahead. Furthermore, in a case in which the ECUdetermines that emergency braking is necessary based on this distance measurement result, the ECUcan promptly instruct automatic braking to the vehicle control unit.

6 FIG. In addition, in the image capturing surface phase difference range gate distance measuring mode, the distance may be calculated by dividing the target distance range R ininto, for example, three or more ranges. In that case, the distance may be calculated after compositing image signals having less noise among each range gate image signal. By doing so, the number of distance calculation processing becomes smaller, and the load in distance measurement processing can be reduced.

109 200 200 101 200 8 FIG. In step S, it is determined whether the movable apparatusis in a traveling state, and in a case in which the movable apparatusis in a traveling state, the process proceeds to step S. In a case in which the movable apparatusis not in a traveling state, the processing flow ofis ended.

As described above, in the present embodiment, processing for calculating the distance to the object based on the first photoelectric conversion signal and the second photoelectric conversion signal generated without range gate control is executed based on a determination result of the visibility condition determination unit. Alternatively, switching whether to execute processing for calculating the distance to the object based on the first photoelectric conversion signal and the second photoelectric conversion signal generated by range gate control is performed based on the above-described determination result.

6 FIG. 102 300 300 411 412 It should be noted that, as explained using, with only range gate control of the photoelectric conversion elementand the light emitter, distance measurement of the object can only be performed in the target distance range R. In contrast, in the present embodiment, by performing range gate control of the light emitter, the first photoelectric conversion unit, and the second photoelectric conversion unit, and by performing distance measurement based on the image capturing surface phase difference method, the distance of the object can be accurately calculated in bad weather conditions.

300 102 Hereinafter, an explanation is provided with respect to the Second Embodiment of the present disclosure. It should be noted that, in the First Embodiment, an explanation was provided with respect to a method for accurately measuring distance to an object by performing range gate control of the light emission time of the light emitterand the exposure time and the exposure timing of the photoelectric conversion elementfor image capturing surface phase difference distance measurement.

102 In the Second Embodiment, an explanation is provided with respect to a method for improving parallax reliability accuracy acquired by range gate control of the photoelectric conversion elementfor image capturing surface phase difference distance measurement based on the target distance range R acquired by range gate control.

Parallax reliability is an index indicating how much error is included in a calculated parallax value, and each pixel has parallax reliability. In addition, because a distance value is calculated based on parallax, parallax reliability is also an index indicating how much error is included in the distance value.

For example, a ratio of standard deviation to an average value of signal values included in a matching region can be evaluated as parallax reliability. When a change in signal values (contrast) within the matching region is large, the standard deviation becomes large.

In addition, in a case in which an amount of light incident on a pixel is large, the average value becomes large, and there is more noise. That is, the average value of the signal values has a positive correlation with the amount of noise. The ratio of standard deviation to the average value (standard deviation/average value) corresponds to a ratio of magnitude of contrast to the amount of noise.

If contrast is sufficiently large with respect to the amount of noise, it can be estimated that error in the calculated parallax value is small. That is, the larger the parallax reliability, the smaller the error in the calculated parallax value, and the parallax value can be said to be more accurate.

3 FIG. 8 FIG. 108 It should be noted that a functional block diagram in the Second Embodiment has the same configuration as, and the present embodiment is preferably implemented after step Sin. However, the implementation is not limited thereto.

11 FIG. 11 FIG. 105 is a flowchart showing an example of parallax reliability improvement processing in a distance measuring method according to the Second Embodiment. It should be noted that operations of each step of the flowchart ofare sequentially performed by a CPU and the like serving as a computer in the camera control unitexecuting a computer program stored in the memory.

201 105 105 103 102 105 8 FIG. In step S, the camera control unitdetermines whether or not the distance measurement processing of the First Embodiment is in the image capturing surface phase difference range gate distance measuring mode. Specifically, the camera control unitdetermines whether the process proceeded to step Safter determining in step Softhat bad weather conditions were not present ahead of the vehicle, or proceeded to step Safter determining that bad weather conditions were present ahead of the vehicle.

201 202 11 FIG. In a case in which it is determined in step Sthat the process is not in the image capturing surface phase difference range gate distance measuring mode, the processing flow ofis ended, and in a case in which it is determined that the process is in the image capturing surface phase difference range gate distance measuring mode, the process proceeds to step S.

202 In step S, it is determined whether or not distance measurement results match range gate distance information. It should be noted that each pixel in a first range gate image signal (A), a second range gate image signal (A), a first range gate image signal (B), and a second range gate image signal (B) has target distance range R information.

104 730 Accordingly, the distance measuring unitdetermines whether or not target distance range R information possessed by each pixel and distance values possessed by pixels corresponding to an object of the composite imagein the image capturing surface phase difference range gate distance measuring mode are in agreement. It should be noted that agreement in this context is determined as agreement even without complete agreement if values are within a predetermined allowable range.

12 FIG.A 12 FIG.B andare diagrams for explaining a method of determining whether target distance range R information in a range gate image signal according to the Second Embodiment and a distance measurement result of an object according to the image capturing surface phase difference range gate distance measuring mode are in agreement.

12 FIG. 730 By using, an explanation is provided with respect to a method of determining whether or not target distance range R information in each pixel of each range gate image signal and distance measurement values in pixels corresponding to objects of the composite imagein the image capturing surface phase difference range gate distance measuring mode are in agreement.

12 FIG.A 202 730 shows a case in which, in step S, target distance range R information in each pixel of each range gate image signal and distance measurement values in pixels corresponding to objects of the composite imagein the image capturing surface phase difference range gate distance measuring mode are in agreement.

12 FIG.A 10 FIG.D 12 FIG.A 730 730 is the composite imageshown in. In, each pixel in the composite imagecontains distance information of target distance ranges R 0 m to 30 m and 30 m to 60 m by range gate control, and distance measurement values of an object calculated by the image capturing surface phase difference range gate distance measuring mode.

12 FIG.A 830 820 For example, in, the distance measurement value of an object in the image capturing surface phase difference range gate distance measuring mode is calculated as 5 m for the pedestrianwith respect to the target distance range 0 m to 30 m. In addition, the distance measurement value of the vehicleis calculated as 40 m with respect to the target distance range 30 m to 60 m. Accordingly, the distance measurement values are in agreement with each target distance range R information.

830 820 830 820 Thereby, the possibility that the pedestrianexists at 5 m and the vehicleexists at 40 m is high, and parallax reliability representing errors in distances of the pedestrianand the vehiclecan be estimated to be high.

730 203 In a case in which target distance range R information in each pixel of each range gate image signal and distance measurement values in pixels corresponding to objects of the composite imagein the image capturing surface phase difference range gate distance measuring mode are in agreement, the process proceeds to step S.

203 104 730 730 In step S, the distance measuring unitincreases parallax reliability in the composite image. Methods of increasing a parallax reliability value in the composite imageinclude adding a constant X greater than 0 to the parallax reliability, or multiplying by a variable Y greater than 1. However, the methods are not limited thereto.

In this manner, in the present embodiment, parallax reliability is updated based on the distance to the object calculated based on the first photoelectric conversion signal and the second photoelectric conversion signal generated by performing range gate control.

12 FIG.B 202 730 shows a case in which, in step S, target distance range R information in each pixel of each range gate image signal and distance measurement values in pixels corresponding to objects of the composite imagein the image capturing surface phase difference range gate distance measuring mode are not in agreement.

12 FIG.B 12 FIG.A 10 FIG.D 12 FIG.B 730 730 is, similar to, the composite imageof. In, each pixel in the composite imagecontains distance information of target distance ranges R 0 m to 30 m and 30 m to 60 m by range gate control and distance information of objects calculated by the image capturing surface phase difference range gate distance measuring mode.

12 FIG.B 830 820 For example, in, the distance measurement result of an object by the image capturing surface phase difference range gate distance measuring mode is 40 m for the pedestrianwith respect to the target distance range 0 m to 30 m. In addition, the distance measurement value of the vehicleis calculated as 20 m with respect to the target distance range 30 m to 60 m, and the distance measurement values are not in agreement with each target distance range R information.

830 820 830 820 Accordingly, the possibility that the pedestrianexists at 5 m and the vehicleexists at 40 m is low, and parallax reliability representing errors in distances of the pedestrianand the vehiclecan be estimated to be low.

730 204 In a case in which target distance range R information in each pixel of each range gate image signal and distance measurement values in pixels corresponding to objects of the composite imagein the image capturing surface phase difference range gate distance measuring mode are not in agreement, the process proceeds to step S.

204 104 730 730 In step S, the distance measuring unitdecreases parallax reliability in the composite image. Methods of decreasing a parallax reliability value in the composite imageinclude subtracting a constant X greater than 0 from the parallax reliability, or multiplying by a variable Y less than 1. However, the methods are not limited thereto.

102 In this manner, determination is made as to whether or not distance measurement values acquired by range gate control of the photoelectric conversion elementfor image capturing surface phase difference distance measurement are in agreement with target distance range R acquired by range gate control. Then, by changing the parallax reliability according to the result, the accuracy of parallax reliability can be improved.

102 102 Hereinafter, an explanation is provided with respect to the Third Embodiment of the present disclosure. In the First Embodiment, an explanation was provided with respect to a configuration in which the photoelectric conversion elementis a CMOS sensor. In the Third Embodiment, an explanation is provided with respect to a configuration in which the photoelectric conversion elementincludes an avalanche photodiode (hereinafter, “APD”).

102 102 The photoelectric conversion elementincluding the APD can digitally count the number of incident photons and output the count value from the pixel as a photoelectrically converted digital signal. Unlike CMOS sensors, the photoelectric conversion elementincluding the APD can quickly switch the exposure time ON/OFF.

102 13 FIG. 16 FIG. In addition, because the APD has no readout noise, the original signal does not deteriorate even when read out multiple times with a single accumulation. An explanation is provided with respect to the configuration and operation of the photoelectric conversion elementthat is provided with the APD by usingto.

13 FIG. 102 102 911 921 is a diagram showing a configuration example of the photoelectric conversion elementaccording to the Third Embodiment. Hereinafter, an explanation is provided with respect to an example of a photoelectric conversion element having a so-called stacked structure in which the photoelectric conversion elementhas two substrates, a sensor substrateand a circuit substrate, that are stacked and electrically connected.

911 912 921 922 912 However, the photoelectric conversion element may have a so-called non-stacked structure in which configurations included in the sensor substrate and configurations included in the circuit substrate are arranged in a common semiconductor layer. The sensor substrateincludes a pixel region. The circuit substrateincludes a circuit regionthat processes signals detected in the pixel region.

14 FIG. 102 102 102 910 910 is a schematic diagram showing a configuration example of the photoelectric conversion elementincluding the APD, and is a plan view of the photoelectric conversion elementviewed from a light incident direction. The photoelectric conversion elementis configured by arranging a plurality of 2 row x 2 column pixel groupsin a matrix. The pixel grouphas four infrared pixels IR that detect IR (infrared) light.

910 300 910 It should be noted that the arrangement of each pixel in the pixel groupis not limited thereto, and may be changed according to a wavelength band of light emitted from the light emitter. That is, the pixel groupmay be configured by, for example, an infrared pixel IR that detects IR light, a red pixel that detects red color, a green pixel that detects green light, and a blue pixel that detects blue light.

910 920 In addition, the pixel groupmay be configured by two green pixels that detect green color, a red pixel that detects red color, and a blue pixel that detects blue light. In a case in which the above-described pixel arrangement is used, the two green pixels are arranged diagonally. In addition, each pixel is provided with a photoelectric conversion unit.

15 FIG. 14 FIG. 921 921 1060 1120 1150 1100 1130 1110 1140 is a diagram showing a configuration example of the circuit substrate. The circuit substrateincludes a signal processing circuitconfigured to process charge photoelectrically converted by the pixels shown in, a readout circuit, a control pulse generation unit, a horizontal scanning circuit, a vertical signal line, a vertical scanning circuit, and an output circuit.

1110 1150 1110 The vertical scanning circuitreceives a control pulse supplied from the control pulse generation unit, and sequentially supplies a control pulse to a plurality of pixels arranged in a row direction on a row-by-row basis. A logic circuit such as a shift register and an address decoder is used for the vertical scanning circuit.

920 1060 1060 A signal output from the photoelectric conversion unitof each pixel is processed in each signal processing circuit. The signal processing circuitis provided with a counter, a memory, and the like, and a digital value is held in the memory.

1100 1060 The horizontal scanning circuitinputs to the signal processing circuita control pulse that sequentially selects each column in order to read out a signal from the memory of each pixel in which a digital signal is held.

1130 1060 1110 1130 102 1120 1140 1060 1130 Signals are output to the vertical signal linefrom the signal processing circuitsof pixels in a row selected by the vertical scanning circuit. The signal output to the vertical signal lineis output to the outside of the photoelectric conversion elementvia the readout circuitand the output circuit. The signal processing circuitincorporates a plurality of buffers connected to the vertical signal line.

13 FIG. 15 FIG. 922 912 1110 1100 1120 1140 1150 912 911 As shown inand, in a plan view, the circuit regionis arranged in a region overlapping the pixel region. Then, in a plan view, the vertical scanning circuit, the horizontal scanning circuit, the readout circuit, the output circuit, and the control pulse generation unitare arranged so as to overlap a peripheral region outside of the pixel regionof the sensor substrate.

911 912 912 1110 1100 1120 1140 1150 That is, the sensor substrateincludes the pixel regionand a non-pixel region arranged around the pixel region. Then, in a plan view, the vertical scanning circuit, the horizontal scanning circuit, the readout circuit, the output circuit, and the control pulse generation unitare arranged in a region overlapping the non-pixel region.

1130 1120 1140 1130 1120 1130 15 FIG. It should be noted that arrangement of the vertical signal lineand the arrangement of the readout circuitand the output circuitare not limited to the example shown in. For example, the vertical signal linemay be arranged extending in the row direction, and the readout circuitmay be arranged at the terminal end of the vertical signal line.

1060 In addition, the signal processing circuitneed not necessarily be provided one-to-one for each photoelectric conversion unit, and may instead be configured such that one signal processing unit is shared among a plurality of photoelectric conversion units and performs sequential signal processing.

16 FIG. 13 FIG. 15 FIG. 1060 1060 910 is a diagram showing an example of an equivalent circuit of the signal processing circuitaccording to the Third Embodiment, and shows an example of an equivalent circuit of the signal processing circuitcorresponding to each pixel of the pixel groupinand.

16 FIG. 16 FIG. It should be noted thatexplains one APD, and the first photoelectric conversion unit and the second photoelectric conversion unit of the First Embodiment are each assumed to be configured by an APD. That is, two instances of the configuration shown inare arranged for each pixel.

That is, the first photoelectric conversion unit and the second photoelectric conversion unit are each provided with an APD configured to generate a pulse according to photons, a counter that counts a number of pulses, and a memory that stores a count value of the counter.

2010 920 2010 The APDincluded in the photoelectric conversion unitgenerates an electron-hole pair according to incident light by photoelectric conversion. One of two nodes of the APDis connected to a power supply line to which a driving voltage VL (first voltage) is supplied.

2010 2020 In addition, the other of the two nodes of the APDis connected to a power supply line to which a driving voltage VH (second voltage) higher than the driving voltage VL is supplied via a quench element.

16 FIG. 2010 2010 2010 In, one node of the APDis an anode, and the other node of the APD is a cathode. A reverse bias voltage such that the APDperforms avalanche multiplication operation is supplied to the anode and the cathode of the APD.

By establishing a state in which such a voltage is supplied, charge generated by incident light causes avalanche multiplication, and avalanche current is generated.

It should be noted that, in a case in which a reverse bias voltage is supplied, there is a Geiger mode that operates at a voltage difference larger than a breakdown voltage between the anode and the cathode, and a linear mode that operates at a voltage difference in the vicinity of the breakdown voltage or at a voltage difference equal to or less than the breakdown voltage.

An APD operated in the Geiger mode is called an SPAD. In the case of an SPAD, for example, the driving voltage VL (first voltage) is −30 V, and the driving voltage VH (second voltage) is 1 V. It should be noted that an SPAD is included as a type of APD.

1060 2020 2100 2110 2120 2020 2010 The signal processing circuitincludes the quench element, a wave shaping unit, a counter circuit, and a memory circuit. The quench elementis connected to a power supply line to which the driving voltage VH is supplied and to one of the nodes, the anode or the cathode, of the APD.

2020 2010 The quench elementfunctions as a load circuit (quench circuit) during signal amplification by avalanche multiplication, and has a function of suppressing avalanche multiplication by suppressing a voltage supplied to the APD(quench operation).

2020 2010 In addition, the quench elementhas a function of returning a voltage supplied to the APDto the driving voltage VH by flowing a current corresponding to a portion that dropped in voltage during the quench operation (recharge operation).

16 FIG. 1060 2100 2110 2120 2020 An example is shown inin which the signal processing circuitincludes the wave shaping unit, the counter circuit, and the memory circuitin addition to the quench element.

2100 2010 2100 The wave shaping unitshapes a voltage change of the cathode of the APDobtained upon detection of a photon, and outputs a pulse signal. For example, an inverter circuit is used as the wave shaping unit.

16 FIG. 2100 In, an example using one inverter as the wave shaping unitis shown, and a circuit in which a plurality of inverters are connected in series may be used, or another circuit having a waveform shaping effect may be used.

2110 2100 2130 2110 2110 The counter circuitcounts a number of pulses output from the wave shaping unit, and holds a count value. In addition, when a control pulse RES is supplied via a drive line, a signal held in the counter circuitis reset. Here, the counter circuitgenerates a signal based on a difference between count values at a start time and an end time of an accumulation period.

2120 1110 2140 2120 1130 15 FIG. 15 FIG. A control pulse SEL is supplied to the memory circuitfrom the vertical scanning circuitofvia a drive line(not shown in), and electrical connection and non-connection between the memory circuitand the vertical signal lineare switched.

2120 2110 1130 2120 The memory circuitfunctions as a memory configured to temporarily store a count value of the counter, and output an output signal from the counter circuitof the pixel to the vertical signal linevia the memory circuit.

2020 2010 920 1060 920 It should be noted that a switch such as a transistor may be arranged between the quench elementand the APD, or between the photoelectric conversion unitand the signal processing circuit, and electrical connection may be switched. Similarly, supply of the driving voltage VH or the driving voltage VL supplied to the photoelectric conversion unitmay be electrically switched by using a switch such as a transistor.

17 FIG. 2010 2100 0 1 2010 is a diagram schematically showing a relationship of operation and output signal of the APD. An input side of the wave shaping unitis denoted as “node A”, and an output side is denoted as “node B”. In a period from time tto time t, a potential difference of VH-VL is applied to the APD.

2010 1 2010 2020 When a photon is incident on the APDat time t, avalanche multiplication occurs in the APD, and avalanche multiplication current flows to the quench element, and voltage of node A drops.

2010 2010 2 When the amount of voltage drop becomes larger, and the potential difference applied to the APDbecomes smaller, the avalanche multiplication of the APDstops as occurs at time t, and the voltage level of node A no longer drops below a certain value.

2 3 3 2100 Thereafter, in a period from time tto time t, current that compensates for the voltage drop from the driving voltage VL flows to node A, and at time t, node A settles to the original potential level. At this time, a portion at which an output waveform at node A falls below a threshold value is wave-shaped by the wave shaping unit, and is output as a pulse signal at node B. The above is an explanation with respect to the configuration and operation of the APD.

102 300 In this manner, in the present embodiment, by performing range gate control of the photoelectric conversion elementincluding the APD and the light emitter, a range gate image signal having less noise compared to a CMOS sensor can be acquired.

Accordingly, in the image capturing surface phase difference range gate distance measurement described in the First Embodiment, by measuring distance to an object based on a range gate image having less noise, calculation of a distance value of higher accuracy becomes possible.

102 300 102 In addition, by performing range gate control of the photoelectric conversion elementincluding the APD and the light emitter, because the exposure time width of the photoelectric conversion elementcan be controlled so as to be shorter, the target distance range R can be made smaller compared to a CMOS sensor. That is, by making the target distance range R smaller, distance resolution can be increased, and distance accuracy can be improved.

200 It should be noted that, in the above-described embodiment, although the movable apparatuswas explained by using an example of a vehicle such as an automobile and the like, the movable apparatus may be any object capable of movement, such as an aircraft, a train, a ship, a drone, an AGV, a robot, and the like.

While the present disclosure has been described with reference to embodiments, it is to be understood that the disclosure is not limited to the disclosed embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

In addition, as a part or the whole of the control according to the embodiments, a computer program realizing the function of the embodiments described above may be supplied to the distance measuring apparatus and the like through a network or various storage media. Then, a computer (or a CPU, an MPU, or the like) of the distance measuring apparatus and the like may be configured to read and execute the program. In such a case, the program and the storage medium storing the program configure the present disclosure.

In addition, the present disclosure includes those realized using at least one processor or circuit configured to perform functions of the embodiments explained above. For example, a plurality of processors may be used for distribution processing to perform functions of the embodiments explained above.

This application claims the benefit of Japanese Patent Application No. 2024-153685, filed on Sep. 6, 2024, which is hereby incorporated by reference herein in its entirety.