An information processing apparatus that communicates with a display device that has an image capturing unit and that generates an image to be displayed on the display device, a composed image in which a CG (Computer Graphics) object in a virtual space according to the position and orientation of the image capturing unit is overlaid on a portion of an image, in real space, captured by the image capturing unit. An image generating unit performs a contact determination between the CG object in the virtual space and an operation unit in the real space based on a thickness model generated by a thickness model generating unit, and generates, as an image to be displayed on the display device, the composed image to which the result of the contact determination is reflected.
Legal claims defining the scope of protection, as filed with the USPTO.
1. An information processing apparatus that communicates with a display device that has an image capturing unit and that generates, as an image to be displayed on the display device, a composed image in which a computer graphics (CG) object in a virtual space according to the position and orientation of the image capturing unit is overlaid on a portion of an original image, in real space, captured by the image capturing unit, the information processing apparatus comprising: at least one processor operatively coupled to a memory, serving as: (a) an image acquisition unit configured to acquire an original image captured by the image capturing unit; (b) a setting unit configured to set information related to a thickness of an operation unit that a user operates in the real space; (c) a surface model generating unit configured to detect an area in which the operation unit is present in the original image acquired by the image acquisition unit, and to generate a surface model of the operation unit that is visible from the image capturing unit based on the area; (d) a thickness model generating unit configured, based on the information related to the thickness set by the setting unit, to generate a thickness model that has a thickness from the surface model, wherein the thickness model generating unit comprises a deforming unit configured to deform the thickness model so that the thickness model does not protrude outside of an area that the operation unit, detected from the original image in the real space, indicates when the thickness model is projected onto the area, and the deforming unit deforms, in a line of sight direction of the image capturing unit, an area of the model that is outside of the area; and (e) an image generating unit configured to perform a contact determination between the CG object in the virtual space and the operation unit in the real space based on the thickness model generated by the thickness model generating unit, and to generate, as the image to be displayed on the display device, the composed image to which the result of the contact determination is reflected.
2. The apparatus according to claim 1 , wherein the thickness model generating unit comprises a thickness adding unit configured to add thickness in a normal direction to the surface model.
3. The apparatus according to claim 1 , wherein the thickness model generating unit has a computing unit configured to compute a thickness from a volume.
4. The apparatus according to claim 1 , wherein the thickness model generating unit has a computing unit configured to compute a thickness from an original image obtained by capturing the operation unit.
5. The apparatus according to claim 1 , wherein the display unit is a head mounted display (HMD).
6. The apparatus according to claim 5 , wherein the apparatus is a part of a system that presents, to a user wearing the HMD, the composed image in which the CG object is overlaid to the image captured by the image capturing unit in real time.
7. The apparatus according to claim 1 , wherein the operation unit is a hand of the user.
8. A method of controlling an information processing apparatus that communicates with a display device that has an image capturing unit and that generates, as an image to be displayed on the display device, a composed image in which a computer graphics (CG) object in a virtual space according to the position and orientation of the image capturing unit is overlaid on a portion of an original image, in real space, captured by the image capturing unit, the method comprising: acquiring an original image captured by the image capturing unit; setting information related to a thickness of an operation unit that a user operates in the real space; detecting an area in which the operation unit is present in the acquired original image, and generating a surface model of the operation unit that is visible from the image capturing unit based on the area; generating a thickness model that has a thickness from the surface model based on the information related to the set thickness, wherein the generating of the thickness model comprises deforming the thickness model so that the thickness model does not protrude outside of an area that the operation unit, detected from the original image in the real space, indicates when the thickness model is projected onto the area, and the deforming the thickness model deforms, in a line of sight direction of the image capturing unit, an area of the model that is outside of the area; and performing a contact determination between the CG object in the virtual space and the operation unit in the real space based on the generated thickness model, and generating, as the image to be displayed on the display device, the composed image to which the result of the contact determination is reflected reflected.
9. A non-transitory computer-readable storage medium storing a program which, when read and executed by a computer, causes the computer to perform steps in a method of controlling an information processing device that communicates with a display device that has an image capturing unit and that generates, as an image to be displayed on the display device, a composed image in which a computer graphics (CG) object in a virtual space according to the position and orientation of the image capturing unit is overlaid on a portion of an original image, in real space, captured by the image capturing unit, the method comprising: acquiring an original image captured by the image capturing unit; setting information related to a thickness of an operation unit that a user operates in the real space; detecting an area in which the operation unit is present in the acquired original image, and generating a surface model of the operation unit that is visible from the image capturing unit based on the area; generating a thickness model that has a thickness from the surface model, based on the information related to the set thickness, wherein the generating of the thickness model comprises deforming the thickness model so that the thickness model does not protrude outside of an area that the operation unit, detected from the original image in the real space, indicates when the thickness model is projected onto the area, and the deforming the thickness model deforms, in a line of sight direction of the image capturing unit, an area of the model that is outside of the area; and performing a contact determination between the CG object in the virtual space and the operation unit in the real space based on the generated thickness model, and generating, as the image to be displayed on the display device, the composed image to which the result of the contact determination is reflected.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
August 24, 2018
June 2, 2020
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.