A view extrapolation apparatus and a view extrapolation method to generate images at a plurality of virtual points using a relatively small number of input images are disclosed. The view extrapolation apparatus and the view extrapolation method output a view at a reference point, the view at the reference point being formed of frames according to time, generating the frames of the view at the reference point to generate a resized frame, and generating an extrapolated view at a virtual point using the resized frame.
Legal claims defining the scope of protection, as filed with the USPTO.
1. An image apparatus, comprising: an input view generation unit to output a view at a reference point, the view at the reference point being formed of reference frames according to time; an importance calculation unit to calculate an importance of each of a plurality of regions forming the reference frames; a resizing unit to resize the reference frames based on the importance and to generate a resized frame; and an extrapolated view generation unit to generate an extrapolated view at a virtual point using the resized frame.
2. The image apparatus of claim 1 , wherein the image apparatus obtains a plurality of views at a plurality of points, and the input view generation unit outputs a view obtained at a leftmost or rightmost point among the plurality of points.
3. The image apparatus of claim 1 , wherein each of the regions comprises color information about pixels included in each of the regions, and the importance calculation unit calculates the importance of each of the regions based on the color information about each of the regions.
4. The image apparatus of claim 3 , wherein the importance calculation unit calculates the importance based on a gradient of the color information.
5. The image apparatus of claim 1 , wherein each of the regions comprises depth information about pixels included in each of the regions, and the importance calculation unit calculates the importance based on the depth information.
6. The image apparatus of claim 1 , wherein each of the regions comprises information about whether pixels included in each of the regions are a foreground or a background, and the importance calculation unit calculates the importance based on the information about whether the pixels are the foreground or the background.
7. The image apparatus of claim 1 , wherein the regions are formed by vertically dividing the reference frames.
8. The image apparatus of claim 7 , wherein the importance calculation unit calculates the importance based on a horizontal position of the regions in the reference frames.
9. The image apparatus of claim 1 , wherein the resizing unit resizes the reference frames by selecting at least one region in ascending order of importance among the plurality of regions and resizing a horizontal size of the selected at least one region.
10. The image apparatus of claim 1 , wherein the plurality of regions are vertical lines of the reference frames.
11. The image apparatus of claim 10 , wherein the resizing unit resizes the reference frames by selecting at least one vertical line in ascending order of importance among the vertical lines and placing the same vertical line as the selected at least one vertical line next to the selected at least one vertical line.
12. The image apparatus of claim 11 , wherein the resizing unit excludes a vertical line positioned within a predetermined distance from the selected vertical line from a subsequent selection.
13. The image apparatus of claim 1 , wherein the extrapolated view generation unit generates the extrapolated view by transferring a position of a pixel in an integrated frame using color information and depth information about the integrated frame.
14. The image apparatus of claim 13 , wherein the extrapolated view generation unit generates the extrapolated view by transferring the position of the pixel using a weighting proportionate to a distance between the reference point and the virtual point.
15. A method of generating an extrapolated view, comprising: generating an input view to output a view at a reference point, the view at the reference point being formed of reference frames according to time; calculating an importance of each of a plurality of vertical lines forming the reference frames; resizing the reference frames based on the importance and generating a resized frame; and generating an extrapolated view at a virtual point using the resized frame.
16. The method of claim 15 , wherein the calculating of the importance calculates the importance of each of the vertical lines based on at least one of a horizontal position of each of the vertical lines in the reference frames, color information about pixels included in each of the vertical lines, depth information about the pixels, and information about whether the pixels are a foreground or a background.
17. The method of claim 15 , wherein the resizing generates the resized frame by selecting part of the plurality of vertical lines and placing the same vertical lines as the selected vertical lines next to the selected vertical lines.
18. The method of claim 16 , wherein the resizing comprises arranging the plurality of vertical lines in ascending order of importance; selecting a first vertical line in the arrangement; and placing the same vertical line as the selected vertical line next to the selected vertical line, wherein a vertical line positioned within a predetermined distance from the selected vertical line is excluded in the arrangement so that the vertical line is excluded from the selection.
19. The method of claim 15 , wherein the generating of the extrapolated view comprises warping an image to transfer a position of a pixel in the resized frame using a weighting proportionate to a distance between the reference point and the virtual point, color information about the resized frame, and depth information about the resized frame.
20. A non-transitory computer-readable medium comprising a program for instructing a computer to perform the method of claim 15 .
21. A method of producing an extrapolated image from an input image having an image capture view point, comprising: producing a motion estimate estimating a motion of the input image; producing an integrated image from at least one time wise successive image associated with the input image and the input image when the motion estimate is greater than or equal to a motion threshold; producing a resized image from the input image resized responsive to importance of regions of the input image when the motion estimate is less than the motion threshold; and producing the extrapolated image having a virtual view point different from the capture view point from one of the integrated image and the resized image.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
July 7, 2011
January 21, 2014
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.