Mixture of volumetric primitives for efficient neural rendering

Abstract

A method for training a real-time, modeling for animating an avatar for a subject is provided. The method includes collecting multiple images of a subject. The method also includes selecting a plurality of vertex positions in a guide mesh, indicative of a volumetric primitive enveloping the subject, determining a geometric attribute for the volumetric primitive including a position, a rotation, and a scale factor of the volumetric primitive, determining a payload attribute for each of the volumetric primitive, the payload attribute including a color value and an opacity value for each voxel in a voxel grid defining the volumetric primitive, determining a loss factor for each point in the volumetric primitive based on the geometric attribute, the payload attribute and a ground truth value, and updating a three-dimensional model for the subject. A system and a non-transitory, computer-readable medium storing instructions to perform the above method are also provided.

What is claimed is: 
     
       1. A computer-implemented method, comprising:
 collecting multiple images of a subject, the images of the subject including one or more different angles of view of the subject; 
 selecting a plurality of vertex positions in a guide mesh, indicative of multiple vertices of a one or more volumetric primitives enveloping the subject; 
 determining a geometric attribute for each of the one or more volumetric primitives, the geometric attribute including a position, a rotation, and a scale factor of the one or more volumetric primitives; 
 determining a payload attribute for each of the one or more volumetric primitives, the payload attribute including a color value and an opacity value for each voxel in a voxel grid defining the one or more volumetric primitives; 
 determining a loss factor for each point in the one or more volumetric primitives based on the geometric attribute, the payload attribute and a ground truth value; and 
 updating a three-dimensional model for the subject according to the loss factor, the three-dimensional model including the one or more volumetric primitives. 
 
     
     
       2. The computer-implemented method of  claim 1 , wherein selecting the plurality of vertex positions in the guide mesh comprises selecting a constraining factor so that a volume of the one or more volumetric primitives is greater than a selected threshold. 
     
     
       3. The computer-implemented method of  claim 1 , wherein selecting the plurality of vertex positions in the guide mesh comprises selecting a minimum volume value of the one or more volumetric primitives so that each point in the images of the subject is within the one or more volumetric primitives. 
     
     
       4. The computer-implemented method of  claim 1 , wherein the one or more volumetric primitives are minimally overlapping and dynamically moving, and determining the geometric attribute for each of the one or more volumetric primitives comprises allowing a change in the position, the rotation, and the scale factor of the one or more volumetric primitives to reduce the loss factor. 
     
     
       5. The computer-implemented method of  claim 1 , further comprising determining the color value and the opacity value for each voxel by tracing a ray of points for each of the volumetric primitives and accumulating three projected color values and a projected opacity value from the images of the subject along a selected point of view. 
     
     
       6. The computer-implemented method of  claim 1 , wherein determining the payload attribute further comprises determining an opacity fade factor to avoid opacity artifacts in overlapping volume primitives close to a boundary of the one or more volumetric primitives. 
     
     
       7. The computer-implemented method of  claim 1 , wherein determining the loss factor comprises determining a mesh reconstruction loss based on the vertex positions in the guide mesh and a ground truth position on a tracked mesh. 
     
     
       8. The computer-implemented method of  claim 1 , further comprising selecting a number of volumetric primitives and a number of voxels per volumetric primitive based on the loss factor. 
     
     
       9. The computer-implemented method of  claim 1 , further comprising interpolating the three-dimensional model between two key frames of a video capture providing the multiple images of the subject. 
     
     
       10. The computer-implemented method of  claim 1 , further comprising:
 forming a background model with multiple images excluding the subject; and 
 updating the three-dimensional model for the subject comprises combining the one or more volumetric primitives with the background model. 
 
     
     
       11. A system, comprising:
 a memory storing multiple instructions; and 
 one or more processors configured to execute the instructions to cause the system to:
 collect multiple images of a subject, the images of the subject including one or more different angles of view of the subject; 
 select a plurality of vertex positions in a guide mesh, indicative of multiple vertices of a one or more volumetric primitives enveloping the subject; 
 determine a geometric attribute for each of the one or more volumetric primitives, the geometric attribute including a position, a rotation, and a scale factor of the one or more volumetric primitives; 
 determine a payload attribute for each of the one or more volumetric primitives, the payload attribute including a color value and an opacity value for each voxel in a voxel grid defining the one or more volumetric primitives; 
 determine a loss factor for each point in the one or more volumetric primitives based on the geometric attribute, the payload attribute and a ground truth value; and 
 update a three-dimensional model for the subject according to the loss factor, the three-dimensional model including the one or more volumetric primitives, 
 wherein to select the plurality of vertex positions in the guide mesh, the one or more processors execute instructions to select a constraining factor so that a volume of the one or more volumetric primitives is greater than a selected threshold. 
 
 
     
     
       12. The system of  claim 11 , wherein to select the plurality of vertex positions in the guide mesh, the one or more processors execute instructions to select a minimum volume value of the one or more volumetric primitives so that each point in the images of the subject is within the one or more volumetric primitives. 
     
     
       13. The system of  claim 11 , wherein the one or more volumetric primitives are minimally overlapping and dynamically moving, and to determine the geometric attribute for each of the one or more volumetric primitives, the one or more processors execute instructions to allow a change in the position, the rotation, and the scale factor of the one or more volumetric primitives to reduce the loss factor. 
     
     
       14. The system of  claim 11 , wherein the one or more processors further execute instructions to determine the color value and the opacity value for each voxel by tracing a ray of points for each of the volumetric primitives and by accumulating three projected color values and a projected opacity value from the images of the subject along a selected point of view. 
     
     
       15. The system of  claim 11 , wherein to determine the payload attribute the one or more processors execute instructions to determine an opacity fade factor to avoid opacity artifacts in overlapping volume primitives close to a boundary of the one or more volumetric primitives. 
     
     
       16. A computer-implemented method, comprising:
 collecting a binocular image of a subject; 
 generating a three-dimensional model of the subject including a patch of minimally overlapping volumetric primitives based on two or more different views of the subject from the binocular image; 
 determining a payload attribute for one or more volumetric primitives of the three-dimensional model, the payload attribute including a color value and an opacity value for each voxel in a voxel grid defining the one or more volumetric primitives; 
 determining a loss factor for each point in the one or more volumetric primitives based on a geometric attribute, the payload attribute, and a ground truth value; 
 updating the three-dimensional model for the subject based on the loss factor; and 
 embedding the updated three-dimensional model of the subject in an immersive reality environment, for a real-time application. 
 
     
     
       17. The computer-implemented method of  claim 16 , further comprising adjusting a voxel count for each of the patch of minimally overlapping volumetric primitives based on a latency threshold for the real-time application. 
     
     
       18. The computer-implemented method of  claim 16 , wherein embedding the three-dimensional model of the subject in the immersive reality environment comprises animating the three-dimensional model by allowing a change in the geometric attribute in the patch of minimally overlapping volumetric primitives, according to the loss factor, wherein the geometric attribute includes a position, a rotation, and a scale factor of the one or more volumetric primitives. 
     
     
       19. The computer-implemented method of  claim 16 , wherein embedding the three-dimensional model of the subject in the immersive reality environment comprises convolving a translation, rotation and scale deviation of the patch of minimally overlapping volumetric primitives with a guide mesh selected from a sequence of binocular images of the subject. 
     
     
       20. The computer-implemented method of  claim 16 , wherein embedding the three-dimensional model of the subject in the immersive reality environment comprises interpolating the three-dimensional model between two key frames in a sequence of images of the subject.