Real-time 3D facial animation from binocular video

Abstract

A method for providing real-time three-dimensional facial animation from video is provided. The method includes collecting images of a subject, and forming a three-dimensional mesh for the subject based on a facial expression factor and a head pose of the subject extracted from the images of the subject. The method also includes forming a texture transformation based on an illumination parameter associated with an illumination configuration for the images from the subject, forming a three-dimensional model for the subject based on the three-dimensional mesh and the texture transformation, determining a loss factor based on selected points in a test image from the subject and a rendition of the test image by the three-dimensional model, and updating the three-dimensional model according to the loss factor. A system and a non-transitory, computer-readable medium storing instructions to perform the above method are also provided.

What is claimed is: 
     
       1. A computer-implemented method, comprising:
 collecting multiple images of a subject, the images from the subject comprising one or more simultaneous views from different profiles of the subject; 
 forming a three-dimensional mesh for the subject based on a facial expression code and a head pose of the subject extracted from the images of the subject; 
 forming a texture transformation based on an illumination parameter associated with an illumination configuration for the images from the subject; 
 forming a three-dimensional model for the subject based on the three-dimensional mesh and the texture transformation; 
 determining a loss factor based on selected points in a test image from the subject and a rendition of the test image by the three-dimensional model; and 
 updating the three-dimensional model according to the loss factor, wherein collecting multiple images of the subject comprises selecting the illumination configuration based on a position, an intensity, and a color gamut for each of multiple light sources around the subject. 
 
     
     
       2. The computer-implemented method of  claim 1 , further comprising:
 collecting a binocular image from the subject; 
 obtaining a three-dimensional representation of the subject by applying the three-dimensional model to the binocular image from the subject; and 
 embedding the three-dimensional representation of the subject in a virtual reality environment in real-time. 
 
     
     
       3. The computer-implemented method of  claim 1 , wherein forming a texture transformation based on an illumination parameter comprises providing the images from the subject under multiple illumination configurations to a low-resolution multilayered network and to a high-resolution multilayered network; and combining an output from the low-resolution multilayered network with an output of the high-resolution multilayered network. 
     
     
       4. The computer-implemented method of  claim 1 , wherein forming the three-dimensional mesh comprises identifying a facial expression of the subject in the images, and associating a facial expression factor with the facial expression of the subject. 
     
     
       5. The computer-implemented method of  claim 1 , wherein forming the three-dimensional mesh comprises identifying a head pose of the subject, the head pose including a rotation of a head of the subject and a translation of the head of the subject. 
     
     
       6. The computer-implemented method of  claim 1 , wherein forming the texture transformation comprises using a bias matrix and a gain matrix including the facial expression factor code, the head pose, and the illumination parameter. 
     
     
       7. The computer-implemented method of  claim 1 , wherein forming the texture transformation comprises determining an illumination parameter based on an illumination configuration for the images from the subject. 
     
     
       8. The computer-implemented method of  claim 1 , wherein determining a loss factor comprises projecting a three-dimensional representation of the subject onto a two-dimensional image and comparing a selected point in the two-dimensional image with a corresponding point in the test image. 
     
     
       9. The computer-implemented method of  claim 1 , wherein updating the three-dimensional model comprises evaluating the loss factor for an incremental change to the head pose over an incremental period of time. 
     
     
       10. The computer-implemented method of  claim 1 , wherein updating the three-dimensional model according to the loss factor comprises embedding a statistical value for the illumination parameter in the texture transformation, the statistical value derived from a multilayered network comprising the images of the subject under multiple illumination configurations. 
     
     
       11. A system, comprising:
 a memory storing multiple instructions; and 
 one or more processors configured to execute the instructions to cause the system to: 
 collect multiple images of a subject, the images from the subject comprising one or more simultaneous views from different profiles of the subject; 
 form a three-dimensional mesh for the subject based on a facial expression code and a head pose of the subject extracted from the images of the subject; 
 form a texture transformation based on an illumination parameter associated with an illumination configuration for the images from the subject; 
 form a three-dimensional model for the subject based on the three-dimensional mesh and the texture transformation; 
 determine a loss factor based on selected points in a test image from the subject and a rendition of the test image by the three-dimensional model; and 
 update the three-dimensional model according to the loss factor, wherein to collect multiple images of the subject the one or more processors execute instructions to select the illumination configuration based on a position, an intensity, and a color gamut for each of multiple light sources around the subject. 
 
     
     
       12. The system of  claim 11 , further comprising an array of video cameras configured to collect the multiple images of the subject, including one or more simultaneous views from different profiles of the subject. 
     
     
       13. The system of  claim 11 , further comprising an array of illumination sources to adjust the illumination configuration for the images from the subject. 
     
     
       14. The system of  claim 11 , wherein the one or more processors further execute instructions to synchronize the images of the subject collected from two or more different cameras and to form a stereoscopic view of a facial expression of the subject. 
     
     
       15. The system of  claim 11 , further comprising a binocular camera configured to collect a binocular image from the subject, and wherein the one or more processors execute further instructions to:
 obtain a three-dimensional representation of the subject by applying the three-dimensional model to the binocular image from the subject, and 
 embed the three-dimensional representation of the subject in a virtual reality environment in real-time. 
 
     
     
       16. The system of  claim 11 , wherein to update the three-dimensional model the one or more processors are configured to evaluate the loss factor for an incremental change to the head pose of the subject over an incremental period of time. 
     
     
       17. The system of  claim 11 , wherein to determine the illumination parameter the one or more processors are configured to provide the images from the subject under multiple illumination configurations to a low-resolution multilayered network and to a high-resolution multilayered network; and combining an output from the low-resolution multilayered network with an output of the high-resolution multilayered network.