Imaging Apparatus, Control Method for Imaging Apparatus, and Storage Medium

The present disclosure relates to an imaging apparatus, a control method for an imaging apparatus, a computer program and a computer-readable data carrier having stored thereon the said computer program.

A camera intended for handheld use or an information processing apparatus such as a head-mounted display sometimes authenticates a user by comparing a pre-registered image and an image acquired by a camera for capturing the user that is built in the information processing apparatus. In order to correctly perform authentication using images, a person appearing in the pre-registered image and a person appearing in the image acquired by the camera need to be determined to be the same person. Japanese Unexamined Patent Application Publication (Translation of PCT Application) No. 2013-518319 discusses pre-registering a plurality of images each with a different iris size, selecting a registered image with an iris size similar to an iris size of an authentication image at the time of authentication, and calculating a similarity degree of images.

In performing identification of a user using a captured image of an eyeball of the user of an imaging apparatus, further improvement of identification accuracy has been demanded.

The present disclosure is directed to improving the identification accuracy of the user of the imaging apparatus.

According to an aspect of the present disclosure, an imaging apparatus includes an imaging unit configured to capture an eyeball image of a user of the imaging apparatus, a first determination unit configured to determine whether the eyeball image captured by the imaging unit satisfies a first condition, a second determination unit configured to determine whether the eyeball image captured by the imaging unit satisfies a second condition different from the first condition, a first identification unit configured to execute first identification processing for identifying a user of the imaging apparatus based on a first eyeball image determined to satisfy the first condition by the first determination unit, a second identification unit configured to execute second identification processing for identifying the user of the imaging apparatus based on a second eyeball image determined to satisfy the second condition by the second determination unit, and a control unit configured to perform control to record information regarding the user identified by the first identification processing and the second identification processing and the eyeball image captured by the imaging apparatus in a memory in association with each other.

Further features of the present disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings.

Hereinafter, each exemplary embodiment of the present invention will be described with reference to the drawings.

In a first exemplary embodiment, the description will be given of a method of performing registration of an eyeball image in accordance with whether an image captured using a camera built in a viewfinder of an imaging apparatus satisfies a first standard or satisfies a second standard.

are diagrams each schematically illustrating an imaging apparatusaccording to the present exemplary embodiment.is a front perspective view of the imaging apparatus, andis a rear perspective view of the imaging apparatus. In the present exemplary embodiment, as illustrated in, the imaging apparatusis configured such that an imaging lensB is detachably attached to an apparatus main bodyA. In the apparatus main bodyA, a release buttonthat is an operation member for receiving an operation from a user when image capturing of a subject is performed is arranged. In the present exemplary embodiment, a digital still camera will be described as an example of an imaging apparatus, but the imaging apparatus is not limited to the digital still camera and may be a head-mounted display (HMD), a data terminal apparatus equipped with a camera (for example, a smartphone or a tablet personal computer (PC)), or the like.

As illustrated in, a viewfinderis provided on a rear surface of the imaging apparatus. The viewfinderis arranged for the user to look through a display elementto be described below. A detection sensorthat detects the user looking through the viewfinderis also arranged. An eyepiece lens() is arranged inside the viewfinder.

As illustrated in, a rear display deviceand operation buttonsandthat are operation members for receiving various operations are arranged on the rear surface of the imaging apparatus. In the present exemplary embodiment, the rear display deviceis a touch panel display having a function as an operation unit.

is a sectional view of the imaging apparatusthat is taken along a YZ plane extending along a Y-axis and a Z-axis illustrated inand illustrates an internal configuration of the imaging apparatus. Into be described below, corresponding portions are indicated by the same reference numerals.

In, the imaging lensB is attached to the apparatus main bodyA. In the present exemplary embodiment, for the sake of example, the imaging lensB includes two inside lensesand, but may also include a larger number of lenses.

The apparatus main bodyA includes an image sensor. A subject image having passed through the imaging lensB is formed on the image sensor. The apparatus main bodyA also includes a central processing unit (CPU)that controls the entire imaging apparatus, and a memory unitthat stores programs to be executed by the CPU, and various types of data. By predetermined image processing being performed on an image signal obtained from the image sensor, under the control of the CPU, a captured image is generated. The captured image is stored in the memory unit.

The apparatus main bodyA also includes the display element, and a display element drive circuitthat drives the display element. The display elementincludes a liquid-crystal display (LCD), and a captured image from the image sensoris displayed thereon. The eyepiece lensis used to observe an image displayed on the display element, in an enlarged state.

Illumination light sourcesandare light sources for illuminating an eyeballof the user, include infrared light-emitting diodes, are arranged around the eyepiece lens, and emit infrared light. An eyeball image by illumination and images obtained by corneal reflection of the illumination light sourcesandpass through the eyepiece lens, are reflected by an optical splitter, and are formed by a light receiving lenson an eyeball image sensoron which photoelectric conversion elements such as charge-coupled devices (CCD) are two-dimensionally arrayed. The light receiving lensis arranged such that a pupil of the eyeballof the user and the eyeball image sensorare in a conjugate imaging relationship. By predetermined image processing being performed on an image signal obtained from the eyeball image sensor, under the control of the CPU, a captured image of the eyeballis generated. In the present exemplary embodiment, the two illumination light sourcesandare arranged, but three or more illumination light sources may be arranged around the eyepiece lens.

On the other hand, the imaging lensB includes a diaphragm, a diaphragm drive apparatus, a lens drive motor, a lens drive member, and a photocoupler. The lens drive memberincludes a gear. The photocouplerdetects rotation of a pulse platerotating in conjunction with the lens drive member, and conveys the rotation to a focus adjustment circuit.

Based on information regarding a rotation amount of the lens drive memberand information regarding a lens drive amount of the apparatus main bodyA, the focus adjustment circuitdrives the lens drive motorby a predetermined amount to adjust the imaging lensB to an in-focus state. The imaging lensB transmits and receives signals to and from the apparatus main bodyA via a mount contactof the apparatus main bodyA.

is a block diagram illustrating a hardware configuration of the imaging apparatus. The image sensor, the memory unit, eyeball image sensor, and the display element drive circuitare connected to the CPUbuilt in the apparatus main bodyA. By executing a program stored in the memory unit, the CPUimplements processing in a flowchart to be described below. The program includes a program of an image registration application to be described below. The memory unitincludes a read-only memory (ROM), a random access memory (RAM), or the like, and stores a captured image from the image sensor, various types of data to be used in eyeball image registration processing to be described below, and a registered eyeball image. The memory unitfunctions also as a work area during program execution.

The release buttonand the operation buttonsandare also connected to the CPUas operation units, and output information regarding operations received from the user to the CPU. The rear display device, which is also connected to the CPU, displays a captured image from the image sensorand various types of information under the control of the CPU.

The display elementis connected to the display element drive circuit. The CPUcontrols the display element drive circuitand displays a captured image from the image sensorand various types of information on the display element.

The CPUis connected also with the focus adjustment circuitarranged in the imaging lensB and a diaphragm control circuitincluded in the diaphragm drive apparatusvia the mount contact.

A communication interface (I/F)and a storage medium I/Fare also connected to the CPU. The communication I/Fis an interface having at least one of communication configurations of a wired communication configuration and a wireless communication configuration, and performs communication with an external apparatus such as a server via a communication path suitable for the communication configuration to be used. The storage medium I/Fis an interface for connecting with a detachable storage medium, and is a secure digital (SD) card slot, for example. The storage mediumis a storage medium that can be detachably attached to the storage medium I/F, and is an SD card or a universal serial bus (USB) memory, for example. The storage mediumcan be used as a recording destination of a captured image from the image sensorand an eyeball image to be registered in the eyeball image registration processing to be described below.

A line-of-sight direction circuitis also connected to the CPU. The line-of-sight direction circuitperforms analog-to-digital (A/D) conversion on an image signal from the eyeball image sensor, and transmits obtained image information to the CPU. The CPUextracts each feature point of an eyeball image that is necessary for line-of-sight detection in accordance with a predetermined algorithm to be described below, and further calculates a line-of-sight direction of the user from the position of each feature point.

is a block diagram illustrating a functional configuration of the imaging apparatus. By the CPUreading a program stored in the memory unitand executing the program, the imaging apparatusfunctions as an eyeball imaging unit, a determination unit, and an eyeball image registration unit.

The eyeball imaging unitacquires an image using an image signal from the eyeball image sensor. In the present exemplary embodiment, the eyeball imaging unitconsecutively acquires images at a predetermined time interval until a certain period of time elapses, and provides the acquired images to the determination unit.

The determination unitdetermines whether an image acquired from the eyeball imaging unitsatisfies a first standard or satisfies a second standard.

The eyeball image registration unitregisters an image satisfying the first standard as an eyeball image for a first authentication (first authentication eyeball image), and registers an image satisfying the second standard as an eyeball image for a second authentication (second authentication eyeball image). The eyeball image registration unitstores the first authentication eyeball image and the second authentication eyeball image into the memory unitin association with information regarding a user who is using the imaging apparatus(for example, login user).

is a flowchart illustrating processing to be executed in a scene where the imaging apparatusaccording to the present exemplary embodiment performs eyeball image registration. In the following description, by adding “S” to the head of a reference numeral of each process (step), a word “process (step)” is omitted.

In a case where the imaging apparatusis used for the first time, the user of the imaging apparatusperforms the eyeball image registration as advance preparation. After the eyeball image registration is completed, it becomes possible to perform user authentication by first authentication processing and second authentication processing using a pre-registered eyeball image. The first authentication processing is authentication processing that uses a first authentication eyeball image to be described below. The second authentication processing is authentication processing that uses a second authentication eyeball image to be described below.

First, when the power of the imaging apparatusis turned ON, the CPUdisplays a home screen on the rear display device.illustrates an example of the home screen. As illustrated in, an iconfor activating the image registration application, and an iconfor starting image capturing of a subject are displayed. The processing in the flowchart is started in a case where a touch operation is performed on the iconfor activating the image registration application.

In step S, the CPUactivates the program of the image registration application that is stored in the memory unit, and displays an activation screen of the image registration application on the rear display device.illustrates an example of an activation screen to be displayed in this step. As illustrated in, a messageincluding a notification indicating that the image registration application is activated, and an instruction to look through the viewfinderis displayed.

In step S, the CPUdetermines whether an act of looking through the viewfinderhas been detected using a sensor value from the detection sensor. Until the CPUdetects the act of looking through the viewfinder, the CPUstands by in step S, and in a case where the act of looking through the viewfinderis detected (YES in step S), the processing proceeds to step S.

In step S, the CPUdisplay a screen indicating that an image is being acquired on the display elementserving as a viewfinder display device.illustrates an example of a screen to be displayed in step S. As illustrated in, a messageindicating an instruction to perform a look-around to the user is displayed. Here, the look-around refers to an operation of moving a line of sight in such a manner as to view various positions in the image displayed on the display element. The screen illustrated incontinues to be displayed on the display elementduring execution of the registration processing.

In step S, the CPUacquires an image using an image signal from the eyeball image sensor. Hereinafter, the image to be acquired in this step will be referred to as a camera image. During the execution of the registration processing, the CPUconsecutively acquires camera images at a predetermined time interval.

In step S, the CPUexecutes the eyeball image registration processing. Details of the registration processing to be executed in this step will be described below with reference to. The CPUstarts counting of the time lapsed, using a timer, from a start of the registration processing.

In step S, the CPUdetermines whether a certain period of time has elapsed from the start of the registration processing with reference to the timer started in step S. Herein, the CPUdetermines whether ten seconds have elapsed, but a different time may be set in accordance with an environment or a use case. In a case where the CPUhas determined that the certain period of time has elapsed (YES in step S), the processing proceeds to step S. In a case where the CPUhas determined that the certain period of time has not elapsed (NO in step S), the processing returns to step S, and the registration processing is continued.

In step S, the CPUdisplays a screen indicating a lapse of a registration time on the display elementserving as a viewfinder display device, and the processing proceeds to step S.illustrates an example of a screen to be displayed in step S. As illustrated in, a messageincluding an instruction to move away from the viewfinderis displayed.

In step S, the CPUdetermines whether the eyeball image registration is completed. In a case where the CPUdetermines that the eyeball image registration is completed (YES in step S), the CPUdisplays a registration completion screen on the rear display device.illustrates an example of the registration completion screen. Then, the processing in the flowchart ends.

When a touch operation is performed on an iconin the registration completion screen illustrated in, the CPUdisplays a home screen on the rear display device.illustrates an example of the home screen to be displayed in a state in which the eyeball image registration is completed. As illustrated in, an iconfor activating the image registration application and an iconfor starting image capturing of a subject are displayed. If a touch operation is performed on the iconfor starting the image capturing of a subject, the CPUswitches an operation mode to an imaging mode.

On the other hand, in step S, in a case where the CPUdetermines that the eyeball image registration is not completed (NO in step S), the CPUdisplays a registration noncompletion screen on the rear display device.illustrates an example of the registration noncompletion screen. As illustrated in, an iconfor displaying a home screen, and an iconfor restarting image registration are displayed. If a touch operation is performed on the icon, the processing returns to step S, and the CPUrestarts the registration processing. On the other hand, if a touch operation is performed on the icon, the CPUdisplays the home screen on the rear display device. In a case where a touch operation is performed on the iconin step S, since the eyeball image registration is not yet completed, the home screen illustrated inis displayed.

Subsequently, the details of the eyeball image registration processing to be executed in step Swill be described with reference to.is a flowchart illustrating the eyeball image registration processing.

In step S, the CPUacquires one of camera images acquired in step S.illustrates an example of a camera image acquired in this step.

In step S, the CPUdetermines whether an eyeball appears in the camera image acquired in step S. In the present exemplary embodiment, the CPUdetects an eyeball from the camera image. As an eyeball detection method, a method described in “MediaPipe: A Framework for Building Perception Pipelines”,[online],[search on Mar. 19, 2025], <https://arxiv.org/pdf/1906.08172.pdf> may be used, or another method may be used. In a case where the CPUdetermines that eyeball detection is successful (YES in step S), the processing proceeds to step S. In a case where the CPUdetermines that the eyeball detection has failed (NO in step S), the processing proceeds to step Swithout registering the image.

are diagrams each illustrating an example of an eyeball detection result.illustrates an eyeball clipped imageobtained by detecting and clipping an eyeball regionfrom a camera image.illustrates an eyeball clipped imageobtained by detecting and clipping an eyeball regionfrom a camera image. In the present exemplary embodiment, in the case where the CPUdetermines that the eyeball detection is successful, the eyeball clipped imageoris generated from the camera imageoracquired in step S, and the generated image is a target in the registration processing (steps Sand S).

In step S, the CPUdetermines whether the camera image acquired in step Ssatisfies the first standard. In a case where the CPUdetermines that the first standard is satisfied (YES in step S), the processing proceeds to step S. In a case where the CPUdetermines that the first standard is not satisfied (NO in step S), the processing proceeds to step S.

In step S, the CPUdetermines whether the camera image acquired in step Ssatisfies the second standard. In a case where the CPUdetermines that the second standard is satisfied (YES in step S), the processing proceeds to step S. In a case where the CPUdetermines that the second standard is not satisfied (NO in step S), the processing proceeds to step Swithout registering the image.

illustrates an example of data to be used in image determination. Standard dataindicates a condition and a predetermined number of images to be registered (the number of registered images) for each of the first standard and the second standard. Coordinate dataindicates predetermined position coordinates (five points Pto P) in a camera image. The point Pindicates an image center of the camera image. The points Pto Pindicate the vicinities of the four corners of the camera image. The standard dataand the coordinate dataare stored in the memory unitor the like, and read and used by the CPUat the time of image determination. In the present exemplary embodiment, a method of performing a first determination based on the first standard and a second determination based on the second standard will be described, but three or more determinations may be performed using three or more standards. In addition, the predetermined number of images to be registered may be one or two or more.

In the present exemplary embodiment, as the first standard, a condition that a distance between the point Pof the camera image and a pupil center position is a predetermined number of pixels (for example, ten pixels) or less is set. Accordingly, in step S, a camera image in which the pupil center position exists near the point Pis determined to satisfy the first standard. As the second standard, a condition that distances between the points Pto Pof the camera image and the pupil center position are a predetermined number of pixels (for example, ten pixels) or less is set. Accordingly, in step S, a camera image in which the pupil center position exists near the points Pto Pis determined to satisfy the second standard. The above-described predetermined number of pixels may be appropriately changed in accordance with resolution of the camera image or detection accuracy of a pupil center. In the present exemplary embodiment, the predetermined number for the first standard is set to one and the predetermined number for the second standard is set to four, but the predetermined numbers are not limited thereto. The predetermined number for the first standard and the predetermined number for the second standard may be different, or may be the same.

In step S, the CPUregisters the eyeball clipped image generated in step Sas the first authentication eyeball image. Then, the CPUstores the registered first authentication eyeball image in the memory unitin association with information regarding the user who is using the imaging apparatus. Then, the processing proceeds to step S.