A digital medium environment includes at least one computing device (e.g., a user's mobile device) in an environment. Systems and techniques are described for presenting a photorealistic augmented reality experience in real time on the computing device. A lighting representation of the environment is obtained from a partial view of the environment, such as from an image of the environment, and used to render a 3D asset (e.g., graphics of a synthetic object). The 3D asset is inserted into the environment by compositing the rendered 3D asset with the image of the environment accounting for occlusion and based on the lighting representation to form a composite image. Photorealistic processing is applied to the composite image by adding lighting for the rendered 3D asset to a portion of the background of the composite image. Consequently, the inserted graphic is indistinguishable from the real-world scene.
Legal claims defining the scope of protection, as filed with the USPTO.
1. In a digital medium environment to insert rendered assets into an augmented reality experience, a method implemented by a computing device located in an environment, the method comprising: obtaining a three-dimensional (3D) representation of the environment; obtaining an image of the environment; adjusting an adaptive rate based on how quickly lighting in the environment changes; determining, at the adaptive rate, a representation of lighting that represents all the lighting in the environment based on the 3D representation and a partial view of the environment from the image of the environment; rendering a 3D asset based on the representation of lighting to produce a rendered 3D asset including shading at least some pixels of the rendered 3D asset; generating an occlusion mask based on the rendered 3D asset and the image of the environment, the occlusion mask identifying where the rendered 3D asset blocks the environment and where the environment blocks the rendered 3D asset; compositing the rendered 3D asset and the image of the environment to form a composite image based on the occlusion mask and the representation of lighting, the compositing including presenting, in the composite image, the rendered 3D asset where the rendered 3D asset blocks the environment, and the image of the environment where the rendered 3D asset does not block the environment; and adding lighting for the rendered 3D asset to a portion of a background of the composite image.
2. The method as described in claim 1 , wherein the compositing further includes: determining a shadow in the environment caused by the rendered 3D asset; modifying the image of the environment based on the shadow to produce a modified environment image; and presenting the modified environment image in the composite image.
3. The method as described in claim 1 , wherein the image of the environment includes multiple frames and the representation of lighting is determined for each frame of the image of the environment that is captured by the computing device.
4. The method as described in claim 1 , wherein the image of the environment includes multiple frames and the representation of lighting is determined from the multiple frames of the image of the environment.
5. The method as described in claim 4 , wherein at least one of the multiple frames is captured by a different computing device than the computing device and the multiple frames correspond to multiple perspectives of the environment.
6. The method as described in claim 1 , wherein the representation of lighting is determined by obtaining a previously-determined representation of lighting of the environment.
7. The method as described in claim 1 , wherein the determining the representation of lighting includes: receiving an additional representation of lighting from an additional computing device located in the environment; and merging the additional representation of lighting with the representation of lighting determined by the computing device.
8. The method as described in claim 1 , wherein the compositing is done electronically by presenting on a display pixels of the rendered 3D asset where the rendered 3D asset blocks the environment and pixels of the image of the environment elsewhere including where the environment blocks the rendered 3D asset.
9. The method as described in claim 1 , wherein the compositing is done optically with a lens that supports the occlusion mask.
10. The method as described in claim 1 , wherein the adding lighting includes masking the portion of the background and compensating for a halo in the image of the environment around the rendered 3D asset.
11. The method as described in claim 1 , further comprising updating the adaptive rate at which the representation of lighting is determined based on a change in location of the computing device.
12. The method as described in claim 1 , wherein the shading at least some pixels of the rendered 3D asset represents a shadow on the rendered 3D asset cast by the rendered 3D asset.
13. In a digital medium environment, a system comprising: a three-dimensional (3D) data module implemented at least partially in hardware of a computing device to obtain a three-dimensional (3D) representation of the environment; an image capture module implemented at least partially in hardware of the computing device to capture an image of the environment; a lighting representation module implemented at least partially in hardware of the computing device to: adjust an adaptive rate based on how quickly lighting in the environment changes; and determine, at the adaptive rate, a representation of lighting that represents all the lighting in the environment based on the 3D representation and a partial view of the environment from the image of the environment; an asset rendering module implemented at least partially in hardware of the computing device to render a 3D asset based on the representation of lighting to produce a rendered 3D asset including shading at least some pixels of the rendered 3D asset; an occlusion mask generation module implemented at least partially in hardware of the computing device to generate an occlusion mask based on the rendered 3D asset and the image of the environment, the occlusion mask identifying where the rendered 3D asset blocks the environment and where the environment blocks the rendered 3D asset; a compositing module implemented at least partially in hardware of the computing device to composite the rendered 3D asset and the image of the environment to form a composite image based on the occlusion mask and the representation of lighting, wherein said to composite includes to present, in the composite image, the rendered 3D asset where the rendered 3D asset blocks the environment, and the image of the environment where the rendered 3D asset does not block the environment; and a photorealistic processing module implemented at least partially in hardware of the computing device to add lighting for the rendered 3D to a portion of a background of the composite image.
14. The system as described in claim 13 , wherein the representation of lighting is based on a history of images of the environment from the image capture module and 3D representations of the environment from the 3D data module, and indicates where light is coming from in a vicinity of a target insertion point of the rendered 3D asset in the environment.
15. The system as described in claim 13 , wherein the representation of lighting includes individual light sources and a panoramic image that encodes light based on direction.
16. The system as described in claim 13 , wherein the image capture module captures multiple frames of the image of the environment, the compositing module composites the rendered 3D asset and the image of the environment for each frame of the image of the environment, and the lighting representation module determines a new representation of lighting for some but not all of the multiple frames of the image.
17. The system as described in claim 13 , wherein the occlusion mask generation module is configured to generate the occlusion mask by encrypting a sequence of binary indicators that indicate where the rendered 3D asset blocks the environment and where the environment blocks the rendered 3D asset, and the compositing module is configured to decrypt the occlusion mask to form the composite image.
18. In a digital medium environment to insert rendered assets into an augmented reality scene, a method implemented by a first computing device not located in an environment, the method comprising: a step for obtaining a three-dimensional (3D) representation of the environment; a step for receiving an image of the environment from a second computing device located in the environment; a step for adjusting an adaptive rate based on how quickly lighting in the environment changes; a step for determining, at the adaptive rate, a representation of lighting that represents all the lighting in the environment based on the 3D representation and a partial view of the environment from the image of the environment; a step for rendering a 3D asset based on the representation of lighting to produce a rendered 3D asset including shading at least some pixels of the rendered 3D asset; a step for generating an occlusion mask based on the rendered 3D asset and the image of the environment, the occlusion mask identifying where the rendered 3D asset blocks the environment and where the environment blocks the rendered 3D asset; a step for compositing the rendered 3D asset and the image of the environment to form a composite image based on the occlusion mask and the representation of lighting, the compositing including presenting, in the composite image, pixels of the rendered 3D asset where the rendered 3D asset blocks the environment, and pixels of the image of the environment where the rendered 3D asset does not block the environment; a step for adding lighting for the rendered 3D asset to a portion of a background of the composite image; and a step for sending the composite image to the second computing device.
19. The method as described in claim 18 , further comprising: a step for receiving an additional image of the environment from a third computing device located in the environment, wherein the representation of lighting is further based on the additional image; and a step for sending the composite image to the third computing device.
20. The method as described in claim 18 , wherein the adaptive rate is adjusted further based on a complexity of the environment.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
August 11, 2017
September 3, 2019
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.