Image Capturing Apparatus, Image Processing Apparatus, and Method Thereof

1. An image processing apparatus comprising: an image input unit configured to input a plurality of images, each of which corresponds to a different viewpoint, obtained by capturing the images from a plurality of viewpoints; an image generation unit configured to composite the plurality of images to generate a synthetic image with a curved focus surface of the synthetic image, wherein the curved focus surface is set based on focus coordinates of the plurality of images in a three-dimensional space; an acquisition unit configured to acquire focus coordinates representing a plurality of positions to be in focus in the synthetic image; and a surface generation unit configured to generate information representing the curved focus surface from a distance to an object and the acquired focus coordinates, wherein a curvature of the curved focus surface is determined in accordance with a number of the acquired focus coordinates, wherein the image generation unit is configured to composite the plurality of images based on the information representing the curved focus surface, and wherein the image generation unit is implemented using a processor.

2. The apparatus according to claim 1 , wherein the image generation unit generates the synthetic image having focus on the curved focus surface in each pixel of the synthetic image.

3. The apparatus according to claim 1 , wherein the surface generation unit sets a curve having an order corresponding to the number of acquired focus coordinates and passing through the focus coordinates, and expands the curve to obtain the focus surface.

4. The apparatus according to claim 3 , wherein the surface generation unit limits the order not to exceed a predetermined number.

5. The apparatus according to claim 1 , further comprising a depth input unit configured to input a depth of field of the plurality of images, wherein the surface generation unit sets a curve having an order corresponding to a number of groups obtained by grouping the focus coordinates based on the depth of field and passing through representative focus coordinates of the groups, and expands the curve to obtain the focus surface.

6. The apparatus according to claim 1 , further comprising an estimation unit configured to estimate the distance to the object from the plurality of images, wherein the image input unit is further configured to input information representing capturing position of the plurality of images, and the estimation unit estimates the distance to the object from the plurality of images and the information representing capturing positions of the plurality of images.

7. The apparatus according to claim 6 , wherein the image generation unit comprises: a setting unit configured to set a weighting coefficient of each of the plurality of images; and a calculation unit configured to calculate a shift amount of each of the plurality of images based on the information representing the capturing positions and the information representing the curved focus surface, wherein the image generation unit is configured to generate the synthetic image by shifting each of the plurality of images by the shift amount calculated by the calculation unit, multiplying each of the shifted images by the weighting coefficient set by the setting unit, and compositing the plurality of images, each of which is shifted and multiplied according to the shift amount and the weighting coefficient.

8. An image capturing apparatus comprising: a plurality of capturing units which capture an object from a plurality of viewpoints; and an image processing apparatus described in claim 1 .

9. The apparatus according to claim 1 , wherein the curved focus surface is curved in a direction of an axis perpendicular to an image plane of the synthetic image.

10. The apparatus according to claim 1 , wherein the curved focus surface is a surface on which the synthetic image is focused best, and an object not on the curved focus surface is defocused more than an object on the curved focus surface.

11. An image processing method comprising: using a processor to perform the steps of: inputting a plurality of images, each of which corresponds to a different viewpoint, obtained by capturing the images from a plurality of viewpoints; compositing the plurality of images to generate a synthetic image with a curved focus surface of the synthetic image, wherein the curved focus surface is set based on focus coordinates of the plurality of images in a three-dimensional space; acquiring focus coordinates representing a plurality of positions to be in focus in the synthetic image; and generating information representing the curved focus surface from a distance to an object and the acquired focus coordinates, wherein a curvature of the curved focus surface is determined in accordance with a number of the acquired focus coordinates, wherein, in the compositing step, the plurality of images are composited based on the information representing the curved focus surface.

12. A non-transitory computer readable medium comprising a computer-executable program for causing a computer to perform an image processing method, the method comprising the steps of: inputting a plurality of images, each of which corresponds to a different viewpoint, obtained by capturing the images from a plurality of viewpoints; compositing the plurality of images to generate a synthetic image with a curved focus surface of the synthetic image, wherein the curved focus surface is set based on focus coordinates of the plurality of images in a three-dimensional space; acquiring focus coordinates representing a plurality of positions to be in focus in the synthetic image; and generating information representing the curved focus surface from a distance to an object and the acquired focus coordinates, wherein a curvature of the curved focus surface is determined in accordance with a number of the acquired focus coordinates, wherein, in the compositing step, the plurality of images are composited based on the information representing the curved focus surface.

13. An image processing apparatus comprising; an image input unit configured to input a plurality of images, each of which corresponds to a different viewpoint, obtained by capturing the images from a plurality of viewpoints; a surface setting unit configured to set a curved focus surface to be focused on in an image; and an image generation unit configured to composite, based on the curved focus surface set by the surface setting unit, the plurality of images to generate a synthetic image having focus on the curved focus surface, wherein the curved focus surface is set based on focus coordinates of the plurality of images in a three-dimensional space, wherein the surface setting unit comprises: an acquisition unit configured to acquire focus coordinates representing a plurality of positions to be in focus in the synthetic image; and a surface generation unit configured to generate information representing the curved focus surface from a distance to an object and the acquired focus coordinates, wherein a curvature of the curved focus surface is determined in accordance with a number of the acquired focus coordinates, wherein the image generation unit is configured to composite the plurality of images based on the information representing the curved focus surface, and wherein the image generation unit is implemented using a processor.