Mixture of Volumetric Primitives for Efficient Neural Rendering

2. The computer-implemented method of claim 1, wherein selecting the plurality of vertex positions in the guide mesh comprises selecting a constraining factor so that a volume of the one or more volumetric primitives is greater than a selected threshold.

3. The computer-implemented method of claim 1, wherein selecting the plurality of vertex positions in the guide mesh comprises selecting a minimum volume value of the one or more volumetric primitives so that each point in the images of the subject is within the one or more volumetric primitives.

4. The computer-implemented method of claim 1, wherein the one or more volumetric primitives are minimally overlapping and dynamically moving, and determining the geometric attribute for each of the one or more volumetric primitives comprises allowing a change in the position, the rotation, and the scale factor of the one or more volumetric primitives to reduce the loss factor.

5. The computer-implemented method of claim 1, further comprising determining the color value and the opacity value for each voxel by tracing a ray of points for each of the volumetric primitives and accumulating three projected color values and a projected opacity value from the images of the subject along a selected point of view.

6. The computer-implemented method of claim 1, wherein determining the payload attribute further comprises determining an opacity fade factor to avoid opacity artifacts in overlapping volume primitives close to a boundary of the one or more volumetric primitives.

7. The computer-implemented method of claim 1, wherein determining the loss factor comprises determining a mesh reconstruction loss based on the vertex positions in the guide mesh and a ground truth position on a tracked mesh.

8. The computer-implemented method of claim 1, further comprising selecting a number of volumetric primitives and a number of voxels per volumetric primitive based on the loss factor.

9. The computer-implemented method of claim 1, further comprising interpolating the three-dimensional model between two key frames of a video capture providing the multiple images of the subject.

12. The system of claim 11, wherein to select the plurality of vertex positions in the guide mesh, the one or more processors execute instructions to select a minimum volume value of the one or more volumetric primitives so that each point in the images of the subject is within the one or more volumetric primitives.

13. The system of claim 11, wherein the one or more volumetric primitives are minimally overlapping and dynamically moving, and to determine the geometric attribute for each of the one or more volumetric primitives, the one or more processors execute instructions to allow a change in the position, the rotation, and the scale factor of the one or more volumetric primitives to reduce the loss factor.

14. The system of claim 11, wherein the one or more processors further execute instructions to determine the color value and the opacity value for each voxel by tracing a ray of points for each of the volumetric primitives and by accumulating three projected color values and a projected opacity value from the images of the subject along a selected point of view.

15. The system of claim 11, wherein to determine the payload attribute the one or more processors execute instructions to determine an opacity fade factor to avoid opacity artifacts in overlapping volume primitives close to a boundary of the one or more volumetric primitives.

17. The computer-implemented method of claim 16, further comprising adjusting a voxel count for each of the patch of minimally overlapping volumetric primitives based on a latency threshold for the real-time application.

18. The computer-implemented method of claim 16, wherein embedding the three-dimensional model of the subject in the immersive reality environment comprises animating the three-dimensional model by allowing a change in the geometric attribute in the patch of minimally overlapping volumetric primitives, according to the loss factor, wherein the geometric attribute includes a position, a rotation, and a scale factor of the one or more volumetric primitives.

19. The computer-implemented method of claim 16, wherein embedding the three-dimensional model of the subject in the immersive reality environment comprises convolving a translation, rotation and scale deviation of the patch of minimally overlapping volumetric primitives with a guide mesh selected from a sequence of binocular images of the subject.

20. The computer-implemented method of claim 16, wherein embedding the three-dimensional model of the subject in the immersive reality environment comprises interpolating the three-dimensional model between two key frames in a sequence of images of the subject.