Systems and methods enable optimization of a 3D model representation comprising the shape and appearance of a particular 3D scene or object. The opaque 3D mesh (e.g., vertex positions and corresponding topology) and spatially varying material attributes are jointly optimized based on image space losses to match multiple image observations (e.g., reference images of the reference 3D scene or object). A geometric topology defines faces and/or cells in the opaque 3D mesh that are visible and may be randomly initialized and optimized through training based on the image space losses. Applying the geometry topology to an opaque 3D mesh for learning the shape improves accuracy of silhouette edges and performance compared with using transparent mesh representations. In contrast with approaches that require an initial guess for the topology and/or an exhaustive testing of possible geometric topologies, the 3D model representation is learned based on image space differences without requiring an initial guess.
Legal claims defining the scope of protection, as filed with the USPTO.
2. The computer-implemented method of claim 1, wherein the 3D model representation comprises a surface mesh and faces of the surface mesh are opaque.
4. The computer-implemented method of claim 1, wherein the processing, rendering, transporting, and updating comprise a training iteration and further comprising repeating the training iteration for additional 3D topology definitions.
5. The computer-implemented method of claim 4, wherein the 3D topology definition and additional 3D topology definitions are randomly generated for each training iteration.
6. The computer-implemented method of claim 4, wherein at least one of the additional 3D topology definitions is determined based on the probability values.
7. The computer-implemented method of claim 4, wherein the volumetric mesh is subdivided for at least one of the additional 3D topology definitions.
8. The computer-implemented method of claim 1, wherein the rendering and transporting are performed in parallel.
11. The system of claim 10, wherein the 3D model representation comprises a surface mesh and faces of the surface mesh are opaque.
14. The non-transitory computer-readable media of claim 13, wherein the 3D model representation comprises a surface mesh and faces of the surface mesh are opaque.
16. The system of claim 15, further comprising repeating the deforming and the updating for additional 3D topology definitions, wherein the additional 3D topology definitions are different compared with the 3D topology definition.
17. The system of claim 16, wherein the additional 3D topology definitions are defined based on computed probability values for the cells within the volumetric mesh that indicate a likelihood that each cell is intersected by a ray corresponding to the reference image.
18. The system of claim 15, wherein the volumetric mesh is initialized as a uniform 3D grid.
19. The system of claim 15, wherein the volumetric mesh is a dense volumetric mesh.
21. The non-transitory computer-readable media of claim 20, further comprising repeating the deforming and the updating for additional 3D topology definitions, wherein the additional 3D topology definitions are different compared with the 3D topology definition.
22. The non-transitory computer-readable media of claim 21, wherein the additional 3D topology definitions are defined based on computed probability values for the cells within the volumetric mesh that indicate a likelihood that each cell is intersected by a ray corresponding to the reference image.
23. The non-transitory computer-readable media of claim 21, wherein the volumetric mesh is initialized as a uniform 3D grid.
24. The non-transitory computer-readable media of claim 21, wherein the volumetric mesh is a dense volumetric mesh.
Cooperative Patent Classification codes for this invention. Click any code to explore related patents in that topic.
December 13, 2022
May 23, 2023
Browse 5M+ US patents with plain-English claim translations and AI-generated analysis.