Image Processing Apparatus, Image Generating Method, and Storage Medium

1. An image processing apparatus comprising: one or more hardware processors; and one or more memories storing one or more programs configured to be executed by the one or more hardware processors, the one or more programs including instructions for: (A) obtaining object information indicating a three-dimensional position and a three-dimensional shape of a specific object in a three-dimensional space, wherein the specific object is included in at least one of captured images obtained by a plurality of imaging apparatuses; (B) obtaining viewpoint information indicating a three-dimensional position of a virtual viewpoint and a direction from the virtual viewpoint in the three-dimensional space; (C) determining, based on (1) the three-dimensional shape of the specific object indicated by the obtained object information and (2) a positional relationship between (a) the three-dimensional position of the specific object indicated by the obtained object information and (b) the three-dimensional position of the virtual viewpoint indicated by the obtained viewpoint information, a three-dimensional position of a virtual object in the three-dimensional space, wherein the virtual object is not included in the plurality of captured images; (D) determining information presented by the content of the virtual object based on a size of a region corresponding to a virtual object in a virtual viewpoint image corresponding to the obtained viewpoint information; (E) generating the virtual viewpoint image including the region corresponding to a virtual object whose content presents the determined information, based on (i) the obtained object information, (ii) the obtained viewpoint information, and (iii) the determined three-dimensional position of the virtual object.

2. The image processing apparatus according to claim 1 , wherein the three-dimensional position of the virtual object is determined so that an image corresponding to a three-dimensional shape of the specific object and an image corresponding to the virtual object do not overlap in the virtual viewpoint image.

3. The image processing apparatus according to claim 1 , wherein the three-dimensional position in the three-dimensional space is defined by a three-dimensional coordinate system corresponding to a space captured by the plurality of imaging apparatuses.

4. The image processing apparatus according to claim 3 , wherein the three-dimensional position of the virtual object is determined such that a distance between the three-dimensional position of the virtual viewpoint and the three-dimensional position of the virtual object in a case where a distance between the three-dimensional position of the virtual viewpoint and the three-dimensional position of the specific object is a first distance is longer than a distance between the three-dimensional position of the virtual viewpoint and three-dimensional position of the virtual object in a case where the distance between the three-dimensional position of the virtual viewpoint and the three-dimensional position of the specific object is a second distance that is longer than the first distance.

5. The image processing apparatus according to claim 1 , wherein the one or more programs further includes instructions for detecting a movement direction of the specific object; and wherein the three-dimensional position of the virtual object is determined based on the detected movement direction of the specific object.

6. The image processing apparatus according to claim 1 , wherein the one or more programs further includes instructions for specifying a shape of another object that is different from the specific object and is included in the at least one captured images, and wherein the three-dimensional position of the virtual object is determined based on the specified shape of the another object.

7. The image processing apparatus according to claim 1 , wherein the three-dimensional position of the virtual object is determined based on a size of the specific object in the virtual viewpoint image determined according to the obtained object information and the obtained viewpoint information.

8. The image processing apparatus according to claim 1 , wherein at least one of a size of the virtual object and a shape of the virtual object is determined based on the obtained object information and the obtained viewpoint information.

9. The image processing apparatus according to claim 1 , wherein the size of a region corresponding to the virtual object in the virtual viewpoint image is determined based on a distance between (a) the three-dimensional position of the virtual viewpoint indicated by the obtained viewpoint information and (b) the determined three-dimensional position of the virtual object.

10. The image processing apparatus according to claim 9 , wherein where (1) a first case is a case where a distance between the three-dimensional position of the virtual viewpoint indicated by the obtained viewpoint information and the determined three-dimensional position of the virtual object is a first distance and (2) a second case is a case where a distance between the three-dimensional position of the virtual viewpoint indicated by the obtained viewpoint information and the determined three-dimensional position of the virtual object is a second distance that is shorter than the first distance, the following condition is satisfied: information presented by the content of the virtual object in the first case is less than information presented by the content of the virtual object in the second case.

11. The image processing apparatus according to claim 1 , wherein the virtual object is an object for displaying an advertisement in the virtual viewpoint image.

12. The image processing apparatus according to claim 11 , wherein a content of the virtual object displayed in the virtual viewpoint image includes at least one of a company logo, a product logo, and a product image.

13. The image processing apparatus according to claim 1 , wherein the virtual object is an object for displaying, in the virtual viewpoint image, additional information related to a capturing target of the plurality of imaging apparatuses.

14. The image processing apparatus according to claim 13 , wherein a content of the virtual object displayed in the virtual viewpoint image includes at least one of information related to a progress of a capturing-target game, statistical information related to a game, information describing a person, and biometric information of a person.

15. The image processing apparatus according to claim 1 , wherein the generated virtual viewpoint image corresponds to the three-dimensional position of the virtual viewpoint and the direction from the virtual viewpoint indicated by the obtained viewpoint information and includes the virtual object arranged at the determined three-dimensional position of the virtual object.

16. The image processing apparatus according to claim 1 , wherein the specific object is an object that moves in a space captured by the plurality of imaging apparatuses.

17. The image processing apparatus according to claim 16 , wherein the specific object is at least one of a person and a ball in a space captured by the plurality of imaging apparatuses.

18. An image generating method executable by an image processing apparatus, the method comprising: (A) obtaining object information indicating a three-dimensional position and a three-dimensional shape of a specific object in a three-dimensional space, wherein the specific object is included in at least one of captured images obtained by a plurality of imaging apparatuses; (B) obtaining viewpoint information indicating a three-dimensional position of a virtual viewpoint and a direction from a viewpoint in the three-dimensional space; (C) determining, based on (1) the three-dimensional shape of the specific object indicated by the obtained object information and (2) a positional relationship between (a) the three-dimensional position of the specific object indicated by the obtained object information and (b) the three-dimensional position of the virtual viewpoint indicated by the obtained viewpoint information, a three-dimensional position of a virtual object in the three-dimensional space, wherein the virtual object is not included in the plurality of captured images; (D) determining information presented by the content of the virtual object based on a size of a region corresponding to a virtual object in a virtual viewpoint image corresponding to the obtained viewpoint information; and (E) generating the virtual viewpoint image including the region corresponding to a virtual object whose content presents the determined information, based on (i) the obtained object information, (ii) the obtained viewpoint information, and (iii) the determined three-dimensional position of the virtual object.

19. The image generating method according to claim 18 , wherein the three-dimensional position of the virtual object is determined so that an image corresponding to a three-dimensional shape of the specific object and an image corresponding to the virtual object do not overlap in the virtual viewpoint image.

20. A non-volatile computer-readable storage medium which stores a program for causing a computer to execute a method comprising: (A) obtaining object information indicating a three-dimensional position and a three-dimensional shape of a specific object in a three-dimensional space, wherein the specific object is included in at least one of captured images obtained by a plurality of imaging apparatuses; (B) obtaining viewpoint information indicating a three-dimensional position of a virtual viewpoint and a direction from a virtual viewpoint in the three-dimensional space; (C) determining, based on (1) the three-dimensional shape of the specific object indicated by the obtained object information and (2) a positional relationship between (a) the three-dimensional position of the specific object indicated by the obtained object information and (b) the three-dimensional position of the virtual viewpoint indicated by the obtained viewpoint information, a three-dimensional position of a virtual object in the three-dimensional space, wherein the virtual object is not included in the plurality of captured images; (D) determining information presented by the content of the virtual object based on a size of a region corresponding to a virtual object in a virtual viewpoint image corresponding to the obtained viewpoint information; and (E) generating the virtual viewpoint image including the region corresponding to a virtual object whose content presents the determined information, based on (i) the obtained object information, (ii) the obtained viewpoint information, and (iii) the determined three-dimensional position of the virtual object.