Imaging Assistance Device, Imaging Assistance Method, and Storage Medium

The present disclosure relates to a technique of imaging assistance.

Many image capturing apparatuses have a function of focusing by a manual operation. Nevertheless, photographers need some experience and training to focus on a subject as intended in a manual operation. Japanese Patent Application Laid-Open No. 2007-279334 discloses a technique for reducing the difficulty of such manual focusing by superimposing the degree of focus deviation, or the degree of focus accuracy, on the display screen.

Nevertheless, the prior art described in Japanese Patent Laid-Open No. 2007-279334 displays a focus degree of a determined region within a screen, and has an issue that it may be difficult to identify which subject is in-focus information in the screen.

In view of the foregoing, the present disclosure is directed to presenting which subject is in focus to a photographer in an easily understandable manner.

According to an aspect of the present disclosure, an imaging assistance device includes a detection unit configured to detect a plurality of subjects from a captured image obtained by image capturing executed in accordance with an input of an imaging setting related at least to focusing, an estimation unit configured to estimate a defocus range indicating a region of a defocus amount of each subject, a selection unit configured to select at least one subject from among the plurality of detected subjects based on a detection result of each subject and an estimation result of the defocus range of each subject, and an output unit configured to output information regarding the focusing on the selected the subject or subjects.

Features of the present disclosure will become apparent from the following description of embodiments with reference to the attached drawings. The following description of embodiments is described by way of example.

Hereinafter, embodiments will be described with reference to the drawings. Each embodiment and modified examples to be described below are not intended to limit the present disclosure, and not all the combinations of features described in each embodiment are always essential to the solution of the present disclosure. The configuration of each embodiment can be appropriately modified or changed depending on the specification of a device to which the present disclosure is applied, and various conditions (e.g., use condition, use environment). Embodiments and modified examples to be described below can also be partially combined as appropriate. In the following embodiments, the same or similar configurations and processing steps are assigned the same reference numerals, and redundant description will be omitted.

1 FIG. 1 FIG. 2 FIG. 3 6 FIGS.and 1 FIG. 212 is a functional block diagram illustrating a functional configuration example of an imaging assistance device according to the first embodiment. Each functional unit of the imaging assistance device illustrated inis implemented by a system control unitof an image capturing apparatus having a hardware configuration into be described below, executing processing in accordance with flowcharts into be described below. Hereinafter, the overview of each functional unit of an imaging assistance device according to an embodiment of the present embodiment will be described with reference to.

101 A setting unitchanges an imaging setting of the image capturing apparatus based on an imaging setting input by a user (hereinafter, referred to as a photographer). Here, the imaging setting of the image capturing apparatus includes, for example, the setting of an in-focus position, and settings of a diaphragm, a shutter speed, ISO sensitivity, and the like. In the present embodiment, the description will be given of an example case where setting input of an in-focus position (focus position) is performed by a manual operation as an input of an imaging setting by a photographer, which will be described in detail below.

102 A detection unitdetects one or more subjects from an image (hereinafter, referred to as a live view image) displayed based on a captured image captured by the image capturing apparatus. In the present embodiment, a “subject” is assumed to be a word including meanings of the entirety (e.g., person) or a part (e.g., body, head, eye, etc., in the case of person) of the subject. Hereinafter, in the present embodiment, the entirety or a part of the subject will collectively be described as “subject”, but in a case where a specific distinction is required to be made, these will be clearly described as “the entirety of the subject” or “a part of the subject”. The details of subject detection processing will be described below.

103 102 A range estimation unitestimates a defocus range of a subject detected by the detection unitfrom a live view image of the image capturing apparatus. The details of the defocus range and estimation processing of the defocus range will be described below.

104 103 102 A selection unitselects a subject for which information regarding an imaging setting is to be output based on a defocus range estimated by the range estimation unit, from among subjects detected by the detection unit.

In the present embodiment, an example of selecting an in-focus subject (or focused subject) and a subject with a high focus degree during focusing by a manual operation, which will be described in detail below, will be given as subject selection.

105 104 104 An output unitoutputs information regarding an imaging setting, for a subject selected by the selection unit. In the present embodiment, an example will be given of displaying a frame indicating a subject selected by the selection unitin a highlighted manner and superimposing the frame on a live view image, as an output of information regarding an imaging setting. A detail of the example will be described below.

2 FIG. 2 FIG. 2 FIG. 200 200 210 230 211 200 is a block diagram illustrating an example configuration of major hardware of an image capturing apparatusaccording to the present embodiment. The image capturing apparatusexemplified inis, for example, an interchangeable-lens digital camera including a camera main body, and a lens unitof an image capturing optical system for guiding incident light to an image sensor.illustrates a configuration in a case where the image capturing apparatusis an interchangeable-lens digital camera, but the image capturing apparatus is not limited to a digital camera, and can also be an onboard camera of a smartphone, for example.

210 The camera main bodywill now be described.

211 230 211 232 213 231 The image sensoris, for example, a complementary metal-oxide semiconductor (CMOS) image sensor, and converts an optical image formed by the lens unitinto an electric signal. The optical image is formed by an image formed on the image sensorvia a diaphragmand a shutterfrom a light beam incident on an imaging lens.

212 210 212 211 212 211 230 211 212 1 FIG. 3 6 FIGS.and The system control unitincorporates a known central processing unit (CPU) or the like, and controls each component in the camera main body. The system control unitalso includes an image processing unit that performs various types of image processing on an image capture signal obtained by the image sensor. The system control unitalso includes a phase difference autofocus (AF) unit that performs focus detection processing using a phase error detection method based on focus detection image data (signal for phase difference AF) acquired by the image sensorand processed by the image processing unit. More specifically, the image processing unit generates a pair of image data pieces formed by light beams passing through a pair of pupil regions of the imaging optical system in the lens unit, as focus detection image data. The phase difference AF unit detects an out-of-focus amount based on a deviation amount of the pair of image data pieces. In this manner, the phase difference AF unit performs phase difference AF (imaging plane phase difference AF) that is based on output information of the image sensor, without using a dedicated AF sensor. Especially in the case of the present embodiment, the system control unitimplements each functional unit of the imaging assistance device according to the present embodiment that is illustrated in, by executing processing in accordance with the flowcharts into be described below.

214 212 212 214 A memorystores programs, variable numbers, constant number, and the like for operating the system control unit. The programs include a program for a CPU or the like of the system control unitimplementing processing of each flowchart according to the present embodiment, which will be described below. The memoryalso includes an electrically erasable/storable nonvolatile memory, and stores various parameters and setting values such as ISO sensitivity, an image capturing mode, various types of correction data, and the like.

215 210 A power switchswitches power on/off of the camera main body.

216 A mode switching unitis a switch for switching the setting of various image capturing modes such as live view image capturing and moving image capturing.

217 212 A rear monitorincludes a liquid crystal device and a light-emitting diode (LED) that display image capturing information, such as operating states and messages, including characters, images, voice, and the like, in accordance with the execution of programs in the system control unit.

218 217 212 217 212 A touch panelis arranged in a region nearly-equal to a monitor surface of the rear monitor, detects touch of a finger or a pen, and notifies the system control unitof a touch position on the rear monitor. The system control unitexecutes processing of operations or functions associated with such touch positions.

217 219 212 220 Similarly to the rear monitor, a viewfinder display unitdisplays image capturing information and the like in accordance with the execution of programs in the system control unit, and forms an electronic viewfinder (EVF) viewable through an eyepiece lens.

221 212 217 219 221 An eye approaching state detection unitdetects whether a photographer is looking through the EVF. The system control unitselectively displays the above-described image capturing information and the like on the rear monitoror the viewfinder display unitbased on a detection result of a photographer's eye approaching state that is obtained by the eye approaching state detection unit.

230 Next, a configuration of the lens unitwill be described.

210 230 223 223 230 210 The camera main bodyand the lens unitare mechanically and electrically bonded via a lens mount mechanism. The lens mount mechanismhas a configuration that enables the lens unitto be attached to and detached from the camera main body.

230 231 232 233 234 235 231 231 2 FIG. The lens unitincludes the imaging lens, the diaphragm, a focus ring (not illustrated), a lens drive circuit, a diaphragm control circuit, and a lens control unit. For the sake of simplification,illustrates only one lens included in the imaging lens, but the imaging lensmay include a plurality of imaging lens units.

235 230 235 235 The lens control unitcontrols the entire lens unit. The lens control unitincludes a memory (not illustrated) storing various constant numbers, variable numbers, programs, and the like for operating lenses. The lens control unitalso includes a nonvolatile memory (not illustrated) holding the largest/smallest aperture value, focal length, and the like, which are information unique to the lens unit.

233 231 230 235 The lens drive circuitdrives a focus lens or the like that is included in the imaging lensof the lens unit, under the control of the lens control unit.

234 232 235 The diaphragm control circuitdrives the diaphragmunder the control of the lens control unit.

235 212 210 223 200 212 233 235 231 212 233 211 In addition, the lens control unitcommunicates with the system control unitof the camera main bodyvia the lens mount mechanism. For example, in a case where the image capturing apparatusis in an autofocus mode, the system control unitperforms an autofocus operation by controlling the lens drive circuitvia the lens control unit, and driving a focus lens included in the imaging lens. In the case of the autofocus mode, the system control unitperforms an autofocus operation by driving a focus lens by controlling the lens drive circuitbased on a defocus amount calculated with output information of the image sensor.

200 A manual focus mode is also prepared for the image capturing apparatus. The manual focus mode is an image capturing mode in which an autofocus function is disabled. In the case of the manual focus mode, a photographer can perform focus position adjustment of putting a focus on a desired subject, by performing an operation of manually turning the focus ring (not illustrated).

200 212 235 212 For example, in a case where the photographer rotates the focus ring clockwise, the focus lens is moved in accordance with the clockwise rotation of the focus ring, and a focus position is adjusted in a direction in which a farther subject is brought into focus. In contrast, in a case where the photographer rotates the focus ring counterclockwise, the focus lens is moved in accordance with the counterclockwise rotation of the focus ring, and a focus position is adjusted in a direction in which a closer subject is brought into focus. In the image capturing apparatusaccording to the present embodiment, when the focus ring is operated by the photographer, position information on the focus lens moved in accordance with the operation of the focus ring is transmitted to the system control unitvia the lens control unit. The system control unitcan thereby recognize how the focus position has been adjusted, based on the position information on the focus lens.

3 FIG. 3 FIG. 1 FIG. 2 FIG. 3 FIG. 212 200 200 212 is a flowchart illustrating a simple flow of imaging assisting processing to be performed by the system control unitof the image capturing apparatusaccording to the present embodiment in a case where the image capturing apparatusaccording to the present embodiment is in the manual focus mode. Each processing step illustrated in the flowchart inis performed by each functional unit inthat is formed by the system control unitinexecuting a program according to the present embodiment. Hereinafter, the overview of imaging assisting processing to be performed by each functional unit of the imaging assistance device according to the present embodiment will be described with reference to the flowchart in. In the following flowcharts, letter “S” indicates each processing step.

212 200 301 200 200 200 200 301 101 101 200 200 3 FIG. In accordance with an image capturing start instruction from the photographer, the system control unitof the image capturing apparatusaccording to the present embodiment starts processing in step Sand subsequent steps in the flowchart in. Here, an image capturing start refers to, for example, bringing the image capturing apparatusinto an image capturing mode by turning on the power of the image capturing apparatus, and bringing the image capturing apparatusin a sleep state into an image capturing-executable state with half-press of a shutter button or the like. In a case where the image capturing apparatusis an onboard camera of a smartphone, an image capturing start can be activation of a camera application of the smartphone, or the like. In step S, when the setting unitreceives input of an imaging setting from the photographer, the setting unitchanges an imaging setting of the image capturing apparatusin accordance with the received imaging setting input. In the case of the present embodiment, the imaging setting of the image capturing apparatusis assumed to be in-focus position adjustment (focus position adjustment) of focusing on a subject with a manual operation, as described above.

302 102 As processing in step S, the detection unitdetects one or more subjects from a live view image as described below.

303 103 In step S, the range estimation unitestimates a defocus range from a live view image as described below.

304 104 102 103 104 104 As processing in step S, the selection unitselects a subject for which information regarding an imaging setting is to be output, from among one or more subjects detected by the detection unit, based on the defocus range estimated by the range estimation unit. In the present embodiment, because an example of setting an in-focus position with a manual operation serving as an imaging setting is given, the selection unitperforms selection of a subject, which is serving as a subject for which information regarding an imaging setting is to be output, based on the in-focus position set with the manual operation. The selection unitselects an in-focus subject and a subject with a high focus degree, which will be described in detail below.

305 105 104 105 104 Next, as processing in step S, the output unitoutputs information regarding an imaging setting, for the subject selected by the selection unit. As an output of information regarding an imaging setting, the output unitdisplays a frame indicating the subject selected by the selection unitin a highlighted manner and superimposes the frame on the live view image, which will be described in detail below.

306 212 212 306 301 305 306 212 200 200 200 200 3 FIG. Thereafter, as processing in step S, the system control unitdetermines whether to continue image capturing. Unless an image capturing end instruction is issued by the photographer, the system control unitdetermines to continue image capturing (YES in step S), and repeatedly executes the processing in steps Sto S. When an image capturing end instruction is input from the photographer (NO in step S), the system control unitends the processing in the flowchart in. Here, examples of an image capturing end instruction include, for example, turning off the power of the image capturing apparatus, switching the mode of the image capturing apparatusfrom an image capturing mode to a photo view mode, a case where the image capturing apparatusenters the sleep state after the lapse of a preset time or more from a non-operation state in which an operation is not input from the photographer, and the like. In a case where the image capturing apparatusis an onboard camera of a smartphone, an image capturing end can be the end of a camera application of the smartphone, or the like.

212 211 Next, a defocus amount to be calculated by the system control unitusing output information of the image sensor, and a defocus range to be estimated based on the defocus amount will be described.

First of all, a defocus amount will be described.

4 FIG. 230 211 is a diagram illustrating a relationship between a defocus amount d of the imaging optical system of the lens unit, and a phase difference (image shift amount) between a first focus detection signal and a second focus detection signal obtained from output information of the image sensor.

4 FIG. 4 FIG. 401 231 230 400 211 401 401 411 412 421 422 400 400 400 400 421 422 In, a lensindicates the imaging lensof the above-described lens unit. An imaging planeof the image sensoris arranged to be orthogonal to an optical axis of the lens. An exit pupil of the imaging optical system by the lensis divided into two regions corresponding to a first pupil regionand a second pupil region. The defocus amount d becomes d=0 in an in-focus state in which a distance (magnitude) from an image formation position C of light beams from subjectsand, to the imaging planebecomes |d|, and the image formation position C exists on the imaging plane. The defocus amount d is defined such that a front focus state in which the image formation position C exists at a position closer to the subjects than the imaging planeis represented by a negative sign (d<0), and a state in which the image formation position C exists on an opposite side of the subjects across the imaging planeis represented by a positive sign (d>0). In the imaging optical system illustrated in, the subjectis in the in-focus state (d=0), and the subjectis in the front focus state (d<0). In the present embodiment, the front focus state (d<0) and a back focus state (d>0) will be collectively referred to as a defocus state (|d|>0).

411 422 1 1 400 211 322 1 1 400 1 In the case of the front focus state (d<0), light beams having passed through the first pupil regionamong light beams from the subject, are once condensed, and then spread to a width Γwith being centered on a centroid position Gof the light beams, whereby a defocused image is formed on the imaging plane. The defocused image is received by each first focus detection pixel on the image sensor, and a first focus detection signal is generated. That is, the first focus detection signal becomes a signal indicating a subject image in which a subjectis defocused by a defocus width Γat the centroid position Gof light beams on the imaging plane. The defocus width Γof the subject image increases approximately in proportion to an increase in the magnitude |d| of the defocus amount d.

412 422 2 2 400 211 322 2 2 400 2 Similarly, in the case of the front focus state (d<0), light beams having passed through the second pupil region, among light beams from the subject, are once condensed, and then spread to a width Γwith being centered on a centroid position Gof the light beams, whereby a defocused image is formed on the imaging plane. The defocused image is received by each second focus detection pixel on the image sensor, and a second focus detection signal is generated. That is, the second focus detection signal becomes a signal indicating a subject image in which the subjectis defocused by a defocus width Γat the centroid position Gof light beams on the imaging plane. The defocus width Γof the subject image increases approximately in proportion to an increase in the magnitude |d| of the defocus amount d.

1 2 Similarly, in the case of the front focus state (d<0), a magnitude |p| of an image shift amount p (=difference G-Gbetween centroid positions of light beams) between the first focus detection signal and the second focus detection signal also increases approximately in proportion to an increase in the magnitude |d| of the defocus amount d.

In contrast, in the case of the back focus state (d >0), an image shift direction between the first focus detection signal and the second focus detection signal becomes opposite to that in the case of the above-described front focus state.

212 211 In this manner, the magnitude of an image shift amount between the first focus detection signal and the second focus detection signal increases in accordance with an increase in the magnitude of a defocus amount. The system control unitaccording to the present embodiment performs focus detection of an imaging plane phase error detection method of calculating a defocus amount from an image shift amount between the first focus detection signal and the second focus detection signal obtained from output information of the image sensor. In other words, the phase difference AF unit converts an image shift amount into a defocus amount using a conversion coefficient calculated based on a base-line length, from a relationship that the magnitude of an image shift amount between the first focus detection signal and the second focus detection signal increases in accordance with an increase in the magnitude of a defocus amount of an image capture signal. In the present embodiment, a product [Fδ] of an F-value of the diaphragm in an imaging optical system during image capturing and a permissible circle of confusion δ is used as a unit of a defocus amount.

Next, a defocus range will be described.

In the present embodiment, a defocus range refers to a value region of a defocus amount of a subject.

5 5 FIGS.A andB are diagrams to be used for the description of a defocus range.

5 FIG.A 5 FIG.A 5 FIG.A 5 FIG.A 5 FIG.A 511 200 512 200 512 513 200 513 514 200 514 515 200 512 illustrates a state in which an image of a personis captured using the image capturing apparatus. A rangeinindicates the spreading of an object (pupil) in a depth direction when a pupil of the person is viewed from the image capturing apparatus, and the rangecorresponds to a value region of a defocus amount of the pupil of the person (i.e., a defocus range of the pupil). Similarly, a rangeinindicates the spreading of an object (face) in the depth direction when a face of the person is viewed from the image capturing apparatus, and the rangecorresponds to a region of a defocus amount of the face of the person (i.e., a defocus range of the face). Similarly, a rangeinindicates the spreading of an object (body) in the depth direction when a body of the person is viewed from the image capturing apparatus, and the rangecorresponds to a value region of a defocus amount of the body of the person (i.e., a defocus range of the body). An in-focus positioninindicates that an in-focus position of the image capturing apparatuscoincides with the rangeof the pupil of the person.

212 211 103 512 513 514 200 200 521 512 522 513 523 514 2 FIG. 5 FIG.B 5 FIG.A 5 FIG.B 5 FIG.B 5 FIG.B 5 FIG.A 5 FIG.A 5 FIG.A The system control unitincalculates a defocus amount using output information of the image sensor, and the range estimation unitestimates a defocus range based on the defocus amount.is a diagram schematically illustrating defocus ranges estimated respectively for the rangeof the pupil of the person, the rangeof the face of the person, and the rangeof the body of the person, which are exemplified in. A horizontal axis direction inindicates the magnitude of a defocus amount, and a side close to the image capturing apparatusis defined as “close” and a side far from the image capturing apparatusis defined as “far”. A length of a line segment indicated by an arrow inindicates a defocus range serving as a value region of a defocus amount.illustrates a defocus rangeestimated for the rangeof the pupil in, a defocus rangeestimated for the rangeof the face in, and a defocus rangeestimated for the rangeof the body in.

5 FIG.A 5 FIG.B 200 511 511 511 511 523 103 In, for example, regarding the spreading in the depth direction of the body of the person viewed from the image capturing apparatus, a closest side corresponds to a tip of nose of the person, and a farthest side corresponds to a tip of shoulder of the person. Thus, a closest value (largest value) of a defocus amount of the body of the person is a defocus amount indicating the tip of nose of the person, and a farthest value (smallest value) of a defocus amount is a defocus amount indicating the tip of shoulder of the person. That is, a region defined by such closest value and the farthest value of the defocus amount is a defocus range of the body of the person. In, the defocus rangein the body of the person is estimated from the closest value and the farthest value of the defocus amount. In a case where the above-described product [Fδ] of an F-value of the diaphragm in an imaging optical system and a permissible circle of confusion δ is used as a unit of the defocus amount, a defocus range of the body of the person becomes the range of 0.2Fδ to −1.4Fδ when the closest value is 0.2 Fδ and the farthest value is −1.4 Fδ, for example. Subject defocus range estimation processing to be executed by the range estimation unitwill be described in detail below.

6 FIG. 6 FIG. 3 FIG. 6 FIG. 1 FIG. 2 FIG. 212 212 is a flowchart illustrating a detailed flow of imaging assisting processing of displaying an in-focus subject frame in a superimposed manner on a live view image, in accordance with the adjustment of a focus position by a manual operation in the system control unit. In the flowchart in, processing partially overlapping the above-described flowchart inis also illustrated. The processing in the flowchart illustrated inis performed by each functional unit inthat is formed by the system control unitillustrated in, executing a program according to the present embodiment.

212 200 212 601 6 FIG. As described above, when the system control unitof the image capturing apparatusreceives setting input of a focus position change corresponding to a focus ring operation performed by the photographer, in image capturing in the manual focus mode, the system control unitstarts the processing in step Sand subsequent steps in.

601 101 212 In step S, the setting unitof the system control unitreceives imaging setting input corresponding to focus position adjustment corresponding to a focus ring operation performed by the photographer.

602 102 211 102 102 102 211 Next, as processing in step S, the detection unitdetects a subject from an image captured by the image sensorwith a focus position being adjusted by a manual operation of the photographer. Here, in the present embodiment, a subject to be detected is, for example, a person, a dog, a cat, an automobile, or the like. The type of a subject to be detected by the detection unitcan be one type or a plurality of types, or can be a subject preset by the photographer. The detection unitmay detect the entirety of a subject, may detect a part of a subject, or may detect the entirety of a subject and each part of the subject. Examples of a detection method of the entirety of a subject or a part of a subject include a detection method that uses a known machine learning technique. For example, an object detector that can detect an object of a specific type or a part of the object is preliminarily created using a technique such as a convolutional neural network (CNN), and the detection unitdetects a subject by inputting a captured image of the image sensorto the object detector.

7 7 FIGS.A andB are diagrams to be used for the description of an example of subject detection processing.

7 FIG.A 701 700 211 102 702 703 704 705 is a diagram illustrating an example of a detection result in which a body, a face, a right eye, and a left eye of a personare detected from an imagecaptured by the image sensor. The detection unitdetects a subject frame as a detection result of a subject. A subject frameis a frame indicating a detection result of a body. Similarly, a subject frameis a frame indicating a detection result of a face, a subject frameis a frame indicating a detection result of a left eye, and a subject frameis a frame indicating a detection result of a right eye.

7 FIG.B 7 FIG.B 102 711 712 713 714 715 is a table indicating subject information generated from the subject frames detected by the detection unit. In the table in, a type of a subject is described on a column, a part of a subject is described on a column, an upper-left coordinate of a subject frame is described on a column, a lower-right coordinate of a subject frame is described on a column, and the reliability of a detection result is described on a column. The type and the parts of a subject, coordinates of a subject frame, and the reliability of a detection result correspond to information that can be acquired by the above-described object detector that uses a known machine learning technique. In the present embodiment, an example will be described in which a bounding box surrounding a region of a detected subject is set as a subject frame, and coordinates of two points corresponding to the upper left and the lower right of the bounding box are acquired.

602 103 603 103 211 102 211 After the above-described processing in step S, the range estimation unitestimates a defocus range of a subject, as the next processing in step S. The range estimation unitaccording to the present embodiment estimates a defocus range of each subject, based on an image captured by the image sensor, subject information detected by the detection unit, and a map (defocus map) of a defocus amount of a region encompassing each subject. In the present embodiment, the defocus map is map information indicating a defocus amount distribution in which a defocus amount is allocated to each pixel of the entirety or a part of an image captured by the image sensor.

8 8 FIGS.A andB are diagrams illustrating an example of an estimation result of a defocus range.

8 FIG.A 801 811 801 811 800 211 802 803 804 805 812 813 814 815 is a diagram illustrating a detection result in which a personand a dog, and parts of the personand the dogare detected from an imagecaptured by the image sensor. A subject frameis a frame indicating a detection result of a body of a person. Similarly, a subject frameis a frame indicating a detection result of a face of a person, a subject frameis a frame indicating a detection result of a left eye of a person, and a subject frameis a frame indicating a detection result of a right eye of a person. A subject frameis a frame indicating a detection result of a body of a dog, a subject frameis a frame indicating a detection result of a face of a dog, a subject frameis a frame indicating a detection result of a right eye of a dog, and a subject frameis a frame indicating a detection result of a left eye of a dog.

8 FIG.B 8 FIG.A 5 FIG.B 8 FIG.B 8 FIG.B 8 FIG.B 8 FIG.B 8 FIG.B 200 200 821 811 822 823 824 811 831 832 833 834 801 801 811 200 is a diagram illustrating an estimation result of a defocus range of the subject illustrated in. Similarly to the above-described example in, a length of a line segment indicated by an arrow inindicates a defocus range, which is a region of a defocus amount. A horizontal axis direction inindicates the magnitude of a defocus amount, a side close to the image capturing apparatusis defined as “close” and a side far from the image capturing apparatusis defined as “far”.illustrates a defocus rangeestimated for the right eye of the dog. Similarly,illustrates defocus ranges,, andestimated respectively for the left eye, the face, and the body of the dog. Similarly,illustrates defocus ranges,,, andestimated respectively for the right eye, the left eye, the face, and the body of the person. Both ends of each of these line segments indicate a closest value and a farthest value of a defocus amount, and a larger line segment indicates a wider defocus range. For example, for both of the personand the dog, a body is thicker in the depth direction than eyes when viewed from the image capturing apparatus. Thus, a defocus range of a body is wider than defocus ranges of eyes.

Here, a defocus range estimation method will be described in more detail.

A defocus range can be acquired by using a machine learning model machine-learned with input learning data. Examples of a specific algorithm of machine learning include, for example, deep learning of generating a feature amount and a connection weighting coefficient by itself using a neural network. Here, learning that uses a neural network will be described.

Input data to be used for learning includes a learning image, a defocus map, a subject region, and a correct defocus range. In the learning, error detection processing and weight update processing are performed.

In the error detection processing, an error is obtained between training data and output data output from an output layer of a neural network obtained in accordance with input data input to an input layer. The training data at this time is a correct defocus range. In the error detection processing, the difference between the training data and the output data from the neural network can also be calculated with a loss function.

In the weight update processing, an internode connection weighting coefficient of the neural network, and the like are updated such that the error becomes smaller based on an error obtained in error detection processing. In such weight update processing, a connection weighting coefficient and the like are updated using, for example, an error backpropagation method. The error backpropagation method is a method of adjusting an internode connection weighting coefficient of each neural network and the like such that the above-described error becomes smaller. As a result of the above-described learning, a machine learning model that estimates a defocus range is obtained. By using the machine learning model, it is possible to estimate a defocus range. In the present embodiment, an example of estimating a defocus range including a subject region in the input has been described, but a subject region is not essential as the input. After a defocus range is estimated, a corresponding subject region can also be detected.

603 104 604 102 103 104 102 After the above-described processing in step S, the selection unitextracts, as the next processing in step S, candidates of subjects to be highlighted from among subjects detected by the detection unit, based on a defocus range estimated by the range estimation unit. In the present embodiment, the selection unitextracts a subject (A), which can be estimated to be currently in focus, and a subject (B), which can be estimated to have a close in-focus position, from subjects detected by the detection unitas candidates of subjects to be highlighted, based on the estimated defocus range. That is, in the present embodiment, subjects to be eventually highlighted are the subject (A) that can be estimated to be currently in focus and the subject (B) that can be estimated to have a close in-focus position.

9 FIG. The subject (A) that can be estimated to be currently in focus, and the subject (B) that can be estimated to have a close in-focus position will be described with reference to.

9 FIG. 8 FIG.A 8 FIG.B 811 801 is a diagram illustrating an estimation result of a defocus range of each of subjects corresponding to the dogand the personillustrated in, and is a diagram in which explanatory notes are also added to the example infor the description.

9 FIG. 9 FIG. 9 FIG. 904 905 906 907 801 Here, the subject (A) that can be estimated to be in focus is a subject whose current in-focus position falls within the estimated defocus range. When the current in-focus position is represented by a defocus amount in unit of the above-described product [Fδ], which is a product of an F-value of the diaphragm in an imaging optical system and a permissible circle of confusion δ, the current in-focus position becomes 0 Fδ. When a lower limit of a defocus range is equal to or smaller than 0 Fδ, and an upper limit is equal to or larger than 0 Fδ, a focus is put on a subject in the defocus range, accordingly. In other words, when a lower limit and an upper limit of a defocus range are denoted by Fmin and Fmax, respectively, and Fmin≤0 and 0≤Fmax are satisfied, it can be determined that a subject is in focus. In the case of the example illustrated in, defocus ranges,,, andrepresented by line segments of arrows inare defocus ranges of subjects (A) that can be estimated to be in focus. That is, in the case of the subjects exemplified in, the right eye, left eye, face, and body of the personare subjects (A) that can be estimated to be in focus.

9 FIG. 9 FIG. 9 FIG. 900 902 903 811 In contrast, the subject (B) that can be estimated to have a close in-focus position is a subject whose current in-focus position does not fall within an estimated defocus range, but a difference between a lower limit or an upper limit of a defocus range and the in-focus position is equal to or smaller than a threshold value. In other words, the subject (B) that can be estimated to have a close in-focus position is a subject with a high focus degree. When a threshold value is denoted by th, 0<Fmin or Fmax<0 is satisfied, and |Fmin|≤th or |Fmax|≤th is also satisfied, it can be determined that the in-focus position is close. In the case of the example illustrated in, defocus ranges,, andrepresented by line segments of arrows inare defocus ranges of subjects (B) that can be estimated to have a close in-focus position. That is, in the case of the subjects exemplified in, the right eye, face, and body of the dogare subjects (B) that can be estimated to have a close in-focus position.

604 104 605 604 After the above-described processing in step S, the selection unitdetermines, as the next processing in step S, a subject to be actually highlighted, from among candidate subjects to be highlighted that have been extracted in step S. In the present embodiment, by making selection from among a plurality of candidate subjects to narrow down a subject to be actually highlighted, information that is more important for the photographer is preferentially displayed, and display not impairing visibility is enabled.

9 FIG. Determination processing for a subject to be highlighted will be described with reference todescribed above.

104 904 905 906 907 104 910 104 904 905 904 907 801 801 9 FIG. Because highlighting a plurality of parts of the same subject is redundant processing, the selection unitnarrows down a subject to be highlighted, for the defocus ranges,,, andof the above-described subjects (A) that can be estimated to be in focus. For example, the selection unitfocuses attention on a typeof a subject, and selects a part of a subject with a narrowest defocus range among subjects of the same type, as a subject to be highlighted. Nevertheless, in a case where a difference equal to or larger than a predefined threshold value is not included, or the like, it is not necessary to always limit subjects to one, and a plurality of defocus ranges can also be selected. In the case of the example in, the selection unitselects the defocus rangesandserving as narrowest ranges among the defocus rangestoof the person, and determines the right eye and the left eye of the person, which are subjects corresponding to the ranges, as subjects to be highlighted.

10 FIG.A 10 FIG.A 10 FIG.A 1010 1020 1030 1000 211 1010 1020 1030 1010 1020 1030 1011 1010 1012 1010 1021 1020 1022 1020 1031 1030 1032 1030 is a diagram illustrating an example in which a plurality of in-focus subjects (e.g., persons,, and) each includes a body and a face. In, an imagecaptured by the image sensorincludes the plurality of persons,, and.illustrates subject frames indicating detected parts of the persons,, and. A subject frameis a frame indicating a detection result of a body of the person, and a subject frameis a frame indicating a detection result of a face of the person. Similarly, a subject frameis a frame indicating a detection result of a body of the person, a subject frameis a frame indicating a detection result of a face of the person, a subject frameis a frame indicating a detection result of a body of the person, and a subject frameis a frame indicating a detection result of a face of the person.

8 FIG.B 10 FIG.B 10 FIG.A 10 FIG.B 10 FIG.B 1013 1014 1023 1024 1033 1034 1010 1020 1030 Similarly to the above-described example in,is a diagram illustrating an estimation result of a defocus range of a subject illustrated in, and a length of each line segment indicated by an arrow inindicates a defocus range.illustrates defocus rangesand,and, andandestimated for the faces and bodies of the persons,, and, respectively.

1013 1023 1033 104 104 1050 1000 1022 1050 104 605 10 10 FIGS.A andB For example, in a case where a narrower defocus range is selected from among defocus ranges of the same subject (e.g., a face), subject frames of faces corresponding to the defocus ranges,, andare selected and highlighted. Nevertheless, when such plurality of subjects (subject frames of a plurality of faces) is highlighted, visibility from the photographer declines. Thus, the plurality of subjects can be further narrowed down by providing a condition. For example, the selection unitcan preferentially select a subject close to a specific region. In the case of, the selection unitcan use, for example, a centerof the imageas a specific region, and select only the subject frameof the face, which is a subject close to the center. Alternatively, for example, the selection unitcan use a region other than the center of the image, such as a region preliminarily selected by the photographer, as a specific region, and select a subject frame based on a distance from the selected region by the photographer. Because the processing in step Sis processing for enhancing the visibility of subject display in a specific situation where a plurality of subject frames of the same subject exists, for example, the processing can be omitted for the simplification of processing.

605 105 606 104 105 702 102 602 7 FIG.A After the above-described processing in step S, the output unithighlights, as the next processing in step S, a subject selected so far by the selection unitin the processing (step) on a live view image. In the case of the example in the present embodiment, the output unitdisplays a bounding box, such as the subject frameinthat has been detected by the detection unitin step S, in a highlighted manner, and superimposes the bounding box on a subject on a live view image. A method of highlighting a subject is not limited to the display of a bounding box, and can also be a method of extracting an outline of a subject using a known image processing technique, and highlighting the outline, or the like.

105 105 105 The output unitcan also adjust a degree of highlighting based on a focus degree of a subject at the time of highlighting a subject frame. For example, the output unitcan also highlight a subject frame of an in-focus subject most intensively, highlight a subject frame of a subject with a defocus range closer to an in-focus position more intensively, or highlight a subject frame of a subject with a defocus range farther from an in-focus position more weakly. In other words, the output unitcan adjust a degree of highlighting to become higher as a focus degree of a subject becomes higher. Another method of highlighting can also be display or the like with gradually changing a line width, line type, line color, density, transparency, luminance, or the like, for example.

11 11 FIGS.A andB 8 FIG.A 801 811 are diagrams illustrating an example of highlighting a subject example of the personand the dogexemplified in.

11 FIG.A 11 FIG.B 11 FIG.A 11 FIG.A 11 FIG.A 1104 1105 1125 1124 1104 1105 801 1104 1105 illustrates an example in which a subject frame is highlighted.is a diagram illustrating a defocus range estimated for each subject exemplified in. Subject framesandinare frames respectively corresponding to in-focus defocus rangesand. Specifically, the subject framesandare frames indicating the left eye and the right eye of the person.illustrates an example in which the subject framesandare highlighted by drawing the subject frames with solid lines with the thickest line width.

1112 1113 1114 1112 1113 1114 1123 1122 1120 1123 1122 1120 1112 1113 1114 105 1123 1122 1120 1112 1113 1114 11 FIG.B Alternatively, highlighting display can also be performed such that subject frames,, andof subjects that are not in focus but have in-focus positions closer to some extent are gradually highlighted. Defocus ranges respectively corresponding to these subject frames,, andare defocus ranges,, and. Here, distances (differences) between the defocus ranges,, andcorresponding to the subject frames,, and, and their in-focus positions are distances x1, x2, and x3 as illustrated in. A magnitude relation of the distances x1, x2, and x3 is also x1<x2<x3. The output unitcan execute highlighting in a stepwise manner to thicken a line width in accordance with the order of the distances x1, x2, and x3 between the defocus ranges,, andcorresponding to the subject frames,, and, and in-focus positions thereof. In this manner, changing gradually and linearly a highlighting degree based on a distance between an in-focus position and a defocus range makes it easier for the photographer to intuitively recognize a focus degree in focus adjustment.

Examples of gradually changing a thickness of a subject frame in accordance with a distance between an in-focus position and a defocus range include an example of linearly changing the thickness of a subject frame from a thickness wmax with the largest highlighting degree, to a thickness wmin with the smallest highlighting degree. In the case of this example, a thickness w of a subject frame corresponding to a defocus range with the distance x from the in-focus position can be obtained by the following formula (1). In formula (1), a threshold value at which the distance x between an in-focus position and a defocus range is determined to be close is denoted by d, and relationship between the threshold value d and the distance x satisfies 0<x<d.

In the above-described example, the case of changing the thickness of a line has been described, but it is also possible to represent a highlighting degree by changing a line type, for example. Nevertheless, the number of available line types is limited to some extent, and thus, in the case of changing a line type, there is a possibility that the number of change steps of highlight display becomes smaller than that in the case of changing the thickness of a line. In the case of the example of changing a line type, the number of change steps of highlight display can be defined by providing a demarcation to some extent such as using a solid line when 0<(x/d)≤0.2 is satisfied, and using a long dot-dash-line when 0.2<(x/d)≤0.4 is satisfied, for example.

606 212 607 212 607 601 601 605 607 212 6 FIG. After the above-described processing in step S, the system control unitdetermines whether to continue image capturing as the next processing in step S. Unless an image capturing end instruction is issued by the photographer, the system control unitdetermines to continue image capturing (YES in step S), and the processing returns to step Sto repeatedly execute the processing in steps Sto S. When an image capturing end instruction is input from the photographer (NO in step S), the system control unitends the processing in the flowchart in.

200 According to the above-described first embodiment, it is possible to present an in-focus subject to the photographer in an easy-to-understand manner. In the present embodiment, presenting also a subject out-of-focus only by little to the photographer makes it possible to present a subject to be focused, to the photographer in an easy-to-understand manner when the photographer performs adjustment of focusing with a manual operation. In other words, according to the image capturing apparatusaccording to the present embodiment, the convenience improves in a case where the photographer performs focus adjustment by a manual operation in the manual focus mode.

In the above-described first embodiment, an example of changing a highlighting degree according to an in-focus position and a focus degree has been described, but an in-focus position and a focus degree can also be presented by the display or output of means other than a subject frame. In a modified example of the first embodiment, an example of presenting information indicating an in-focus position and a focus degree, to the photographer by the display or output of means other than a subject frame will be described focusing on a difference from the first embodiment.

12 FIG. 12 FIG. 1 FIG. 2 FIG. 1 FIG. 12 FIG. 12 FIG. 6 FIG. 12 FIG. 200 212 601 605 607 105 1206 605 is a flowchart illustrating a flow of processing according to the present modified example of presenting information indicating an in-focus position and a focus degree adapted to focus position adjustment performed by a manual operation, to the photographer by the display or output of means other than a subject frame in the image capturing apparatus. The configuration of each functional unit that executes the imaging assisting processing inis the same as that in the block diagram illustrated in, and thus illustrations and descriptions thereof will be omitted. The system control unitinimplements each functional unit according to this modified example that is illustrated in, by executing processing in accordance with the flowchart in. The processing in steps Sto Sand Sof the flowchart inis the same as the above-described processing of the flowchart in, and thus the description thereof will be omitted. In the case of the flowchart inaccording to this modified example, the output unitperforms processing in step Safter the processing in step S.

13 13 FIGS.A andB are diagrams to be used for the description of imaging assisting processing according to the present modified example.

13 FIG.A 13 FIG.A 1301 1321 1311 1300 211 1302 1301 1322 1321 1312 1311 is a diagram illustrating an example in which a plurality of subjects (personsand, and a dog) appear in an imagecaptured by the image sensor, and subject frames indicating detected parts of the plurality of subjects are displayed. In the example illustrated in, a subject frameof a face of the person, a subject frameof a face of the person, and a subject frameof a face of the dogare detected.

13 FIG.B 13 FIG.A 9 10 FIGS.andB 13 FIG.B 13 FIG.B 13 FIG.B 1331 1332 1333 1311 1301 1321 1332 1301 1331 1333 is a diagram illustrating an estimation result of a defocus range of each subject illustrated in, similarly todescribed above. The length of each line segment indicated by an arrow inindicates a defocus range.illustrates defocus ranges,, andestimated respectively for the faces of the dog, the person, and the person. In the case of the example illustrated in, the defocus rangeis a defocus range including an in-focus position, and thus it can be seen that the face of the personis in focus. In contrast, for example, in the defocus range, it can be seen that a corresponding subject is located in front of an in-focus position. For example, in the defocus range, it can be seen that a corresponding subject is located on the back side of an in-focus position.

12 FIG. 1206 605 105 1302 1312 1322 605 1302 1312 1322 In the flowchart in, when the processing proceeds to step Safter the processing in step S, the output unitdisplays the subject frames,, andto be highlighted that have been determined in the processing up to step S, and superimposes the subject frames on a live view image. Nevertheless, in the case of the present modified example, degrees of highlighting the subject frames,, andare not varied unlike the above-described example in the first embodiment.

105 1207 1341 1342 1343 1341 1302 1301 1342 1312 1311 1343 1322 1321 13 FIG.A 13 FIG.A In the case of the present modified example, the output unitdisplays, as the next processing in step S, images of focus guides,, andillustrated into be superimposed on a live view image, as information indicating an in-focus position and a focus degree. In the example illustrated in, the focus guideindicates an in-focus position and a focus degree in a region indicated by the subject frameof the face of the person. Similarly, the focus guideindicates an in-focus position and a focus degree in a region indicated by the subject frameof the face of the dog, and the focus guideindicates an in-focus position and a focus degree in a region indicated by the subject frameof the face of the person.

14 14 FIGS.A toE 1400 are diagrams to be used for the description of a focus guide.

14 14 FIGS.A toE 1400 illustrate an in-focus position and how to represent a difference in an in-focus position on the focus guide, for the same defocus range.

1400 1402 105 1402 1403 1400 1403 1402 1403 1400 14 14 FIGS.A toE In the focus guideillustrated in, a belt-like indexindicates a defocus range. The output unitdetermines the length of the belt-like indexon a circleof the focus guide(the length of an arc on the circle) in accordance with the magnitude of a defocus amount. In other words, the magnitude of a defocus amount is represented by the length of the arc of the belt-like indexon the circleon the focus guide.

1400 1401 1401 105 1401 1403 1400 1403 1401 1403 1400 105 1401 1403 1400 105 1401 1403 1401 1401 105 1401 1403 1401 1403 14 14 FIGS.A toE In the focus guideillustrated in, triangular indicesindicate in-focus positions, and two triangular indicesare displayed at symmetrical positions to enhance the visibility of the photographer. The output unitdetermines an interval between the two triangular indiceson the circleof the focus guide(the length of the arc on the circle) in accordance with the magnitude of a defocus amount. In other words, the magnitude of a defocus amount is represented by the length of the arc between the two triangular indiceson the circleof the focus guide. The output unitalso changes the position of the two triangular indicesalong the circleof the focus guidein accordance with in-focus position adjustment performed by a manual operation of the photographer. For example, when the photographer operates a focus ring to change a focus position, the output unitchanges the position of the two left and right triangular indicesalong the circlein a direction in which the triangular indicesget closer to each other, or in a direction in which the triangular indicesget away from each other. In a case where an in-focus position is located on a distant side of the center of a defocus range, the output unitarranges the two triangular indiceson the outside of the circle. In other words, it is represented that an in-focus position exists on the distant side of the center of a defocus range by the two triangular indicesexisting on the outside of the circle.

14 FIG.A 1400 1402 105 1401 is a diagram illustrating an example of the focus guidein a case where an in-focus position exists on the distant side of a defocus range indicated by the belt-like index. In a case where the in-focus position exists on the outside of the defocus range, the output unitdisplays the two triangular indicesin white, for example.

14 FIG.B 14 FIG.A 1400 1400 105 1401 is a diagram illustrating an example of the focus guidein a case where an in-focus position moves in a close direction and gets closer to the defocus range more than that of the example of the focus guidein, and the in-focus position falls within the defocus range. In this manner, in a case where the in-focus position falls within the defocus range, the output unitchanges the color of the triangular indicesfrom white to green, for example, to present an in-focus state to the photographer.

14 FIG.C 14 FIG.B 1400 1402 105 1401 1403 is a diagram illustrating an example of the focus guidein a case where an in-focus position further moves in the close direction more than that of the example in, and the in-focus position exists substantially at the center of the defocus range represented by the belt-like index. In a case where the in-focus position exists substantially at the center of the defocus range, the output unitpresents, to the photographer, that the in-focus position exists at the center of the defocus range by displaying the two triangular indicesarranged on both of the outside and inside of the circle.

14 FIG.D 14 FIG.C 14 FIG.D 1400 1402 105 1401 1403 is a diagram illustrating an example of the focus guidein a case where an in-focus position further moves in the close direction more than the example in, and the in-focus position exists on the close direction side of the center of the defocus range represented by the belt-like index. Nevertheless, the example inillustrates that a state in which the in-focus position still exists within the defocus range. In a case where the in-focus position exists on the close direction side of the center of the defocus range and within the defocus range, the output unitdisplays the two triangular indicesto be arranged on the inside of the circle.

14 FIG.E 14 FIG.D 1400 is a diagram illustrating an example of the focus guidein a case where an in-focus position further moves in the closer direction more than that of the example in, and the in-focus position falls outside the defocus range.

105 1401 1401 1403 In a case where the in-focus position moves in the closer direction with respect to the center of the defocus range and falls outside the defocus range, the output unitchanges the color of the two triangular indicesfrom green to white to present an out-of-focus state to the photographer. By arranging the two triangular indicesat this time on the inside of the circle, the photographer can recognize that the in-focus position exists on the close direction side of the center of the defocus range and falls outside the defocus range.

14 14 FIGS.A toE 1402 103 105 1402 105 1402 1401 1402 1403 In, the defocus range indicated by the belt-like indexis illustrated as a constant range without a change to simplify the illustration. However, the range estimation unitactually estimates a defocus range as needed, and the defocus range consequently changes constantly. A change in the defocus range is also caused by a change in focus position made by the photographer, a change in diaphragm, a change in the position of a subject or an image capturing apparatus, a change in subject shape, and the like. For this reason, the output unitactually adjusts the length of the arc of the belt-like indexas needed in accordance with the change in defocus range. At the time, the output unitchanges the length of the arc of the belt-like indexindicating the defocus range, without changing the position of the triangular indexthat indicates an in-focus position. In the present modified example, the magnitude of the defocus amount is represented by the length of the arc of the belt-like indexon the circle, and thus the length of the belt-like index is determined based on a ratio between an in-focus position and a defocus amount at the boundary of the defocus range, with reference to an in-focus position and the center of the defocus range. The position of an index indicating the above-described in-focus position is not moved if the photographer has not moved a focus ring, and if the photographer has moved a focus ring, the position of the index is changed in accordance with a rotation amount. In this manner, in the focus guide according to the present modified example, the display enables the photographer to intuitively feel that the position of the focus ring is the position of an index indicating an in-focus position by giving consistency to the position of an index indicating an in-focus position.

13 FIG.A In the above-described example in, an example in which focus guides are displayed near (e.g., below) all subject frames is illustrated, but the example is not limited to this. For example, one common focus guide can be displayed for a plurality of parts included in the entire one subject, and a plurality of defocus ranges respectively corresponding to a plurality of parts can also be displayed within the one focus guide. This can prevent the occurrence of a state in which a plurality of focus guides is displayed near subject frames, and the visibility of the photographer is impaired, for example.

15 FIG. 15 FIG. 1501 1502 105 1501 1502 is a diagram illustrating an example of displaying a plurality of defocus ranges on one focus guide. In, for example, a belt-like indexindicates a defocus range of a body of a person, and a belt-like indexindicates a defocus range of a part (e.g., face) of the same person. In this case, the output unitdisplays the belt-like indicesandindicating the plurality of defocus ranges, in a distinguishable manner using different colors, transparencies, luminance, or the like, for example.

105 105 The output unitcan also adjust a degree of highlighting a focus guide based on a distance between a defocus range and an in-focus position. By adjusting a degree of highlighting a focus guide, it becomes possible to display a defocus range having a high degree of interest for the photographer, with being highlighted. For example, in a case where there are a subject with a narrow defocus range and a subject with a wide defocus range, it is considered that the subject with a narrow defocus range is more likely to be a subject desired to be focused. For this reason, the output unitcan perform highlight display in a focus guide such that a defocus range of the subject with a narrow defocus range is more highlighted.

105 The output unitcan also adjust the length of a belt-like index indicating a defocus range desired to be displayed, as appropriate depending on whether a position of a subject and an in-focus position exist closer to the close side or the distant side of the center of the defocus range, and the like.

105 Furthermore, the output unitcan also integrate defocus ranges with similar magnitudes into one belt-like index as an attempt for improving visibility. For example, defocus ranges of a right eye and a left eye of a person facing the front side can be collectively displayed as one belt-like index.

1207 105 1208 105 1208 212 607 After the above-described processing in step S, the output unitsends, as the next processing in step S, a feedback indicating that an in-focus position has exceeded the boundary of the defocus range, to the photographer by tactual sense. As an example, when the photographer changes an in-focus position by turning the focus ring, and the in-focus position falls within the defocus range, the output unitnotifies the photographer that the in-focus position falls within the defocus range by slightly vibrating a focus ring. In this manner, using a method other than the highlight display as in the above-described first embodiment to notify the photographer that the in-focus position falls within the defocus range makes it easier for the photographer to recognize a focus degree even in a case where a lot of information is displayed on a screen. The feedback to the photographer by tactual sense can also be performed by the change of rotational resistance of the focus ring, sound output, the vibration of a linked device worn by the photographer, or the like. After the processing in step S, the system control unitadvances the processing to step Ssimilar to the above-described processing.

In the present modified example, an example of transmitting an in-focus position and a focus degree to the photographer by a combination of the screen display of a focus guide and the feedback by tactual sense has been described. However, the transmission to the photographer can also be performed by a method other than the screen display (i.e., using only tactual sense). In this manner, suppressing information display on a screen enables the photographer to capture an image without impairing subject visibility.

As described above, according to the modified example of the first embodiment, it is possible to more intuitively notify the photographer which of the subjects is currently in focus.

As a second embodiment, an example of starting autofocusing based on an in-focus defocus range will now be described mainly focused on a difference from the first embodiment.

16 FIG. 16 FIG. 17 FIG. 16 FIG. 2 FIG. 17 FIG. 17 FIG. 6 FIG. 17 FIG. 101 102 104 103 105 1606 212 601 606 607 1606 1707 606 is a block diagram illustrating a configuration example of each functional unit in an image capturing apparatus according to the second embodiment. As illustrated in, the image capturing apparatus according to the second embodiment includes a setting unit, a detection unit, a selection unit, a range estimation unit, and an output unitthat are similar to those described in the first embodiment, and also includes an AF setting unitin addition to these units.is a flowchart illustrating a flow of processing to be performed by an imaging assistance device according to the second embodiment. Each functional unit inis implemented by the system control unitwith the above-described hardware configuration inexecuting processing in accordance with flowchart illustrated in. In the flowchart in, the processing in steps Sto Sand Sis the same processing as the processing described with reference to. Thus, the description thereof will be omitted. In the case of the flowchart in, the AF setting unitperforms the processing in step Sand subsequent steps after the processing in step S.

1707 1606 1606 1707 1606 1708 1707 1606 607 1708 200 218 As processing in step S, the AF setting unitdetermines whether an autofocus function is enabled by the photographer. In other words, the AF setting unitdetermines whether a change to an autofocus mode from the state in the above-described manual focus mode has been performed. In a case where the autofocus function has been enabled by the photographer (YES in step S), the AF setting unitsubsequently executes the processing in step S. In contrast, in a case where the autofocus function has not been enabled by the photographer (NO in step S), the AF setting unitadvances the processing to step Sby skipping the processing in step S. The enabling of the autofocus function by the photographer can be input made using a dedicated button provided on the image capturing apparatus, or input made using a touch panel, for example, and is not limited.

1708 1606 605 1606 1606 1606 In step S, the AF setting unitsets a target on which focus tracking is to be performed by the autofocus function. A known technique can be used for the autofocusing method, and the description thereof will be omitted here. The target on which focus tracking is to be performed is a subject with the smallest distance between a corresponding defocus range and an in-focus position among subjects determined in step Sto be highlighted. Nevertheless, in a case where there is a plurality of applicable subjects, the AF setting unitselects one from the applicable subjects. In the case of the present embodiment, the AF setting unitselects one subject at random from among a plurality of subjects, for example. Aside from this, the AF setting unitcan preferentially select a subject on which focus tracking can easily be performed, in view of an autofocusing method, for example, based on the size of a subject frame, a variation in defocus amount within a subject region, luminance, or the like.

11 FIG.A 1124 1125 1105 1104 1606 1105 1606 1708 212 607 For example, in the above-described case as illustrated in, defocus ranges with the smallest distance from an in-focus position are the defocus rangesand, which correspond to the subject framesandcorresponding to the eyes of the person. Accordingly, the AF setting unitsets either subject frame of these (e.g., the subject frame) as a tracking target of autofocusing. The AF setting unitthen starts subject tracking by autofocusing. After the processing in step S, the system control unitadvances the processing to the above-described processing in step S.

According to the second embodiment described above, it is possible to select a subject on which a focus is intuitively put by the photographer with a manual operation, also as a subject to be tracked by autofocusing.

The present disclosure can also be implemented by processing of supplying a program for implementing one or more functions of the above-described embodiments, to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus, reading out and executing the program. Alternatively, the present disclosure can also be implemented by a circuit implementing one or more functions (e.g., application specific integrated circuit (ASIC)). All of the above-described embodiments merely indicate examples of embodiment in carrying out the present disclosure, and the technical scope of the present disclosure is not to be construed in a limited manner based on these. In other words, the present disclosure can be executed in various forms without departing from its technical idea or a major feature thereof.

The disclosure of each embodiment includes the following configuration, method, and program.

a detection unit configured to detect a subject from a captured image obtained by image capturing in accordance with an input of an imaging setting related at least to focusing; an estimation unit configured to estimate a defocus range indicating a region of a defocus amount of the subject; a selection unit configured to select at least one subject from among subjects detected by the detection unit, based on a detection result of the subject and an estimation result of the defocus range; and an output unit configured to output information regarding the focusing on the selected subject. An imaging assistance device comprising:

in which the input of the imaging setting is an input of changing a focus position with a manual operation performed by a user, and in which the selection unit selects the subject based on a detection result of the subject, an estimation result of the defocus range, and the focus position. The imaging assistance device according to Configuration 1,

in which, based on a defocus amount of a plurality of parts included in one subject, the estimation unit estimates the defocus range for each of the plurality of the parts, in which the selection unit selects a part with the narrower estimated defocus range from among the plurality of the parts, and in which the output unit outputs the information regarding the focusing on the selected part. The imaging assistance device according to Configuration 1 or 2,

The imaging assistance device according to any one of Configurations 1 to 3, in which the selection unit performs selection of the subject based also on a position of the subject in the captured image in addition to a detection result of the subject and an estimation result of the defocus range.

The imaging assistance device according to any one of Configurations 1 to 4, in which the selection unit performs selection of the subject based on a position of the subject with respect to a specific region in the captured image.

The imaging assistance device according to Configuration 5, in which the selection unit performs selection while prioritizing a subject with a close position with respect to the specific region.

The imaging assistance device according to any one Configurations 1 to 6, in which, as an output of the information regarding the focusing, the output unit displays the subject selected by the selection unit in a highlighted manner.

The imaging assistance device according to any one of Configurations 1 to 7, in which the output unit gradually changes the highlighted display of the subject selected by the selection unit, based on a focus degree on the subject.

The imaging assistance device according to Configuration 8, in which the output unit makes a highlight degree of the highlighted display higher as the focus degree on the subject gets higher.

The imaging assistance device according to any one of Configurations 1 to 6, in which, as an output of the information regarding the focusing, the output unit displays a first index indicating an in-focus position of the selected subject, and a second index indicating an estimation result of the defocus range, near the selected subject.

The imaging assistance device according to Configuration 10, in which the output unit displays an image in which the first index and the second index are arranged in accordance with an arc of a circle.

The imaging assistance device according to Configuration 11, in which the output unit changes a position of the first index with respect to the arc of the circle in accordance with a change in the in-focus position, and changes a length of the second index with respect to the arc of the circle in accordance with a change in the defocus range.

The imaging assistance device according to Configuration 10, in which the output unit displays the first index in a highlighted manner in accordance with a change in the in-focus position, and displays the second index in a highlighted manner in accordance with a change in the defocus range.

The imaging assistance device according to any one of Configurations 10 to 13, in which the output unit displays one set of the first index and the second index for a plurality of parts included in one subject.

The imaging assistance device according to any one of Configurations 1 to 6, in which the output unit outputs the information regarding the focusing as tactual sense to a user.

The imaging assistance device according to Configuration 15, in which the output unit performs the output of the information regarding the focusing by the tactual sense, based on an in-focus position of the selected subject and an estimation result of the defocus range.

The imaging assistance device according to Configuration 16, in which, when the in-focus position falls within the defocus range, the output unit performs the output by the tactual sense.

The imaging assistance device according to any one of Configurations 1 to 17, further comprising a setting unit configured to set an autofocus function with the selected subject being set to a focus tracking target.

detecting a subject from a captured image obtained by image capturing in accordance with an input of an imaging setting related at least to focusing; estimating a defocus range indicating a region of a defocus amount of the subject; selecting at least one subject from among subjects detected by the detecting, based on a detection result of the subject and an estimation result of the defocus range; and outputting information regarding the focusing on the selected subject. An imaging assistance method comprising:

A program that causes a computer to function as the imaging assistance device according to any one of Configurations 1 to 18.

According to the present disclosure, it becomes possible to present an in-focus subject to the photographer.

Embodiment(s) of the present disclosure can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.

While the present disclosure has been described with reference to embodiments, it is to be understood that the present disclosure is not limited to the disclosed embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No. 2024-168723, filed Sep. 27, 2024, which is hereby incorporated by reference herein in its entirety.