Image processing apparatus and method for synthesizing a high-resolution image and a refocused image

1. An image processing apparatus, comprising: a light field data position determiner configured to determine a position of a portion of light field data among light field data corresponding to a scene, according to a desired focusing position; a refocusing image generator configured to generate a refocusing first image by use of the position determined portion of light field data; a high resolution image generator configured to generate a high-resolution image comprising a predetermined setting ratio, relative to the refocusing first image; a synthesis ratio determiner configured to determine a ratio for local synthesis between the high-resolution image and an enlarged image of the refocusing first image, by use of a similarity with respect to the position determined portion of light field data; and an image synthesizer configured to generate a synthesized image by synthesizing the high-resolution image and the enlarged image of the refocusing first image, according to the local synthesis ratio, wherein the synthesis ratio determiner is further configured to determine the ratio for local synthesis, such that the high-resolution image is synthesized with a higher synthesis ratio for a portion comprising a higher similarity with respect to the position-determined portion of light field data than a portion comprising a lower similarity with respect to the position-determined portion of the light field data, and wherein the desired focusing position corresponds to a highest sharpness refocusing image and the refocusing first image corresponds to the highest sharpness refocusing image.

2. The image processing apparatus of claim 1 , wherein the light field data position determiner is further configured to: set a second image, corresponding to a part of an image representing the scene, and comprising the desired focusing position; generate a plurality of refocusing images for the second image while changing a varying alpha value, the varying alpha value representing a relationship between a photographic plane and a refocusing plane; determine a final alpha value, from the varying alpha value, the final alpha value allowing a highest sharpness refocusing image to be generated for the second image; and determine the position of the portion of light field data by use of the determined final alpha value.

3. The image processing apparatus of claim 1 , wherein the synthesis ratio determiner comprises: a non-uniformity map generator configured to generate a non-uniformity map representing a non-uniformity of intensity of light field data at the determined position; a ratio non-uniformity map generator configured to generate a ratio non-uniformity map comprising information about the ratio for local synthesis between the high-resolution image and the enlarged image of the refocusing first image, by use of the non-uniformity map; and a ratio non-uniformity map enlarger configured to enlarge the ratio non-uniformity map at the setting ratio.

4. The image processing apparatus of claim 1 , wherein the image synthesizer is further configured to: enlarge the refocusing first image at the setting ratio; and synthesize the enlarged image of the refocusing first image and the high-resolution image at the determined ratio for local synthesis.

5. The image processing apparatus of claim 2 , wherein the high-resolution image generator comprises: an image frame determiner configured to determine a standard image frame and at least one reference image frames, by use of the position determined portion of light field data comprising the position, the position being determined by the determined final alpha value; a point spread function determiner configured to determine a point spread function, based on an amount of sub-pixel displacement of the at least one reference image frame with respect to the standard image frame; an image interpolator configured to generate a high-resolution standard image frame comprising a higher resolution than the standard image frame by interpolating the standard image frame; and an image restorer configured to restore the high-resolution image by updating the high-resolution standard image frame using the generated high-resolution standard image frame, the point spread function, and the at least one reference image frame.

6. The image processing apparatus of claim 4 , wherein the amount of sub-pixel displacement represents a position difference between light field data corresponding to the standard image frame and light field data corresponding to each reference image frame.

7. The image processing apparatus of claim 4 , wherein the image frame determiner is further configured to: determine a view image seen from an angle as the standard image frame; and determine at least one view image seen from another angle as the reference image frame by use of the position determined portion of light field data comprising the position, the position being determined by the final alpha value.

8. The image processing apparatus of claim 4 , wherein the point spread function determiner is further configured to determine, as the point spread function, a two-dimensional Gaussian function, based on the amount of sub-pixel displacement of the each reference image frame with respect to the standard image frame.

9. The image processing apparatus of claim 4 , wherein the image restorer comprises: a residual value generator configured to generate a residual value by use of the generated high-resolution standard image frame, one of the reference image frames, and a point spread function based on the one reference image frame and the standard image frame; and an image updater configured to update the high-resolution standard image frame by use of the residual value.

10. The image processing apparatus of claim 9 , wherein the residual value comprises a value of the one reference image frame minus a convolution of the high-resolution standard image frame with the point spread function.

11. The image processing apparatus of claim 9 , wherein, if the high-resolution standard image frame is updated, the residual value generator is further configured to generate a residual value by use of the updated high-resolution standard image frame, another one of the at least one reference image frame, and a point spread function based on the another one of the at least one reference image frame and the standard image frame.

12. The image processing apparatus of claim 1 , further comprising a light field data capturer, the light field data capturer comprising: a first optical unit configured to form an image of an object; a photo sensor array configured to capture light rays; and a second optical unit disposed between the first optical unit and the photo sensor array, and configured to direct the light rays toward the photo sensor array by separating the light rays based on the direction of the light rays.

13. An image processing method, the method comprising: determining a position of a portion of light field data among light field data corresponding to a scene, according to a desired focusing position; generating a refocusing first image by use of the position determined portion of light field data; generating a high-resolution image comprising a predetermined setting ratio, relative to the refocusing first image; determining a ratio for local synthesis between the high-resolution image and an enlarged image of the refocusing first image, by use of a similarity with respect to the position determined portion of light field data; and generating a synthesized image by synthesizing the high-resolution image and the enlarged image of the refocusing first image, according to the local synthesis ratio, wherein, in the determining of the ratio for local synthesis, the ratio for local synthesis is determined such that the high-resolution image is synthesized with a higher synthesis ratio for a portion comprising a higher similarity with respect to the position-determined portion of light field data than a portion comprising a lower similarity with respect to the position-determined portion of the light field data, and wherein the desired focusing position corresponds to a highest sharpness refocusing image and the refocusing first image corresponds to the highest sharpness refocusing image.

14. The image processing method of claim 13 , wherein the determining of the position of the portion of light field data comprising: setting a second image corresponding to a part of an image representing the scene and comprising the desired focusing position; generating a plurality of refocusing images for the second image while changing a varying alpha value, the varying alpha value representing a relationship between a photographic plane and a refocusing plane; determining a final alpha value, from the varying alpha value, the final alpha value allowing a highest sharpness refocusing image to be generated for the second image; and determining the position of the portion of light field data by use of the determined alpha value.

15. The image processing method of claim 13 , wherein the determining of the ratio for local synthesis comprises: generating a non-uniformity map representing a non-uniformity of intensity of light field data at the position determined by an alpha value representing a relationship between a photographic plane and a refocusing plane; generating a ratio non-uniformity map including information about the ratio for local synthesis between the high-resolution image and the enlarged image of the refocusing first image by use of the non-uniformity map; and enlarging the ratio non-uniformity map at the setting ratio.

16. The image processing method of claim 13 , wherein the generating of the synthesized image comprises: enlarging the refocusing first image at the setting ratio; and synthesizing the enlarged image of the refocusing first image and the high-resolution image at the determined ratio for local synthesis.

17. The image processing method of claim 14 , wherein the generating of the high-resolution image comprises: determining a standard image frame and at least one reference image frames, by use of the determined alpha value; determining a point spread function, based on an amount of sub-pixel displacement of the at least one reference image frame with respect to the standard image frame; generating a high-resolution standard image frame comprising a higher resolution than the standard image frame, by interpolating the standard image frame; and restoring the high-resolution image by updating the high-resolution standard image frame using the generated high-resolution standard image frame, the point spread function, and the at least one reference image frame.

18. A method of processing an acquired four-dimensional (4D) light field, the method comprising: determining a desired focusing position and estimating an alpha value for the focusing position, the alpha value representing a relationship between a photographic plane and a refocusing plane; generating a refocusing image from angular data according to the alpha value; generating a high-resolution image by increasing a resolution of a focusing region, based on a sub-pixel displacement of the angular data according to the alpha value; dividing the focusing region from an out-of-focus region by checking a similarity of the angular data according to the alpha value; and performing fusion on the refocusing image and the high-resolution image, wherein the high-resolution image is synthesized with a higher synthesis ratio for a portion comprising a higher similarity with respect to a position-determined portion of light field data than a portion comprising a lower similarity with respect to the position-determined portion of the light field data, and wherein the alpha value corresponds to a highest sharpness refocusing image and the refocusing image corresponds to the highest sharpness refocusing image.

19. The method of claim 18 , wherein the alpha value is determined by varying the alpha value until a highest sharpness refocusing image is generated.

20. A non-transitory computer-readable information storage medium storing a program for causing a computer to implement the method of claim 13 .

21. A non-transitory computer-readable information storage medium storing a program for causing a computer to implement the method of claim 18 .

22. An image processing method, comprising: from light field data corresponding to a scene, generating a refocusing image corresponding to a part of an image representing the scene, the refocusing image including a desired focusing position; generating an enlarged image of the refocusing image; generating a high resolution image corresponding to the enlarged image using the light field data corresponding to the part of the image; and generating a synthesized image based on the high resolution image and the enlarged image, wherein the high-resolution image is synthesized with a higher synthesis ratio for a portion comprising a higher similarity with respect to a position-determined portion of light field data than a portion comprising a lower similarity with respect to the position-determined portion of the light field data, and wherein the desired focusing position corresponds to a highest sharpness refocusing image and the refocusing image corresponds to the highest sharpness refocusing image.

23. The method of claim 22 , wherein the refocusing image comprises a highest sharpness among a plurality of images corresponding to the part of the image.