System and method for generating 3-d facial model and animation using one video camera

Abstract

Provided are system and method for generating a 3D facial model and animation using one video camera. The system includes a pre-processing part, a facial model generating part, a transferring part, a projecting part, an error calculating part, and a mesh transforming part. The pre-processing part sets correspondence relations with other meshes, generates an average 3D facial model, and generates a geometrical model and a texture dispersion model. The facial model generating part projects the average 3D facial onto an expressionless facial image frame that stares a front side to generate a performer's 3D facial model. The transferring part transfers a 3D facial model template having an animation-controlled model to the performer's 3D facial model to generate the performer's 3D facial model. The projecting part projects the performer's 3D facial model onto a facial animation video frame including a facial expression. The error calculating part calculates an error projected by the projecting part. The mesh transforming part moves or rotationally converts a joint in such a direction as to minimize the error.

1 . A system for generating a 3 dimension (D) facial model and animation using one video camera, the system comprising:
 a facial model generating part for generating a performer's 3D facial model onto an expressionless facial image frame of facial expression moving images input from a video camera, that stares a front side;   a transferring part for transferring a 3D facial model template having an animation-controlled model to the performer's 3D facial model generated by the facial model generating part to generate the performer's 3D facial model having the animation-controlled model;   a projecting part for projecting the performer's 3D facial model having the animation-controlled model transferred by the transferring part onto a facial animation video frame including a facial expression;   an error calculating part for calculating an error between the projected image from the projecting part and a facial animation video frame including a facial expression; and   a mesh transforming part for translating and rotating a joint in such a direction as to minimize the error calculated by the error calculating part to transform a mesh.   
     
     
         2 . A system of  claim 1 , further comprising a pre-processing part for setting correspondence relations with other meshes with respect to all vertexes of a 3D facial DB, generating an average 3D facial model, and generating variance models of the geometry through principal component analysis (PCA) of a covariance matrix between the average 3D facial model and facial models in DB. 
     
     
         3 . The system of  claim 1 , wherein processes performed by the facial model generating part are replaced by existing 3D facial scanning. 
     
     
         4 . The system of  claim 1 , wherein operations of the error calculating part and the mesh transforming part are repeated for all video frames containing facial expression to generate animation. 
     
     
         5 . The system of  claim 2 , wherein operations of the error calculating part and the mesh transforming part are repeated for all video frames containing facial expression to generate animation. 
     
     
         6 . The system of  claim 3 , wherein operations of the error calculating part and the mesh transforming part are repeated for all video frames containing facial expression to generate animation. 
     
     
         7 . A method for generating a 3 dimension (D) facial model and animation using one video camera, the method comprising the steps of:
 a facial model generating step of generating a performer's 3D facial model by facial model generating part onto an expressionless facial image frame of facial expression moving images input from a video camera, that stares a front side; and   an animation forming step of transferring a 3D facial model template having an animation-controlled model to a performer's 3D facial model generated by a facial model generating part, projecting the performer's 3D facial model having the animation-controlled model to a facial animation video frame including a facial expression to calculate an error, and moving or rotationally converting a joint in such a direction as to minimize the error to transform a mesh.   
     
     
         8 . The method of  claim 7 , wherein the animation forming step comprises:
 projecting the average 3D facial model onto an expressionless facial model frame of moving images including facial expression obtained by a video camera, that stares a front side to generate a performer's 3D facial model;   transferring the 3D facial model template having the animation-controlled model onto the 3D facial model to generate the performer's 3D facial model having the animation-controlled model;   calculating an error between an image generated by projecting the performer's 3D facial model onto a facial animation video frame including facial expression, and a video frame including facial expression;   judging whether the calculated error is located within a minimum error range;   when the calculated error is located within the minimum error range as a result of the judgment, producing 3D animation of facial expression; and   when the calculated error is located outside the minimum error range as a result of the judgment, moving and rotationally converting a joint in such a direction as to minimize the error to transform a mesh.   
     
     
         9 . The method of  claim 7 , further comprising:
 a pre-processing step of setting correspondence relations with other meshes with respect to all vertexes of a 3D facial DB, generating an average 3D facial model, and generating a variance models of the geometry and a texture dispersion model through principal component analysis (PCA) of a covariance matrix between the average 3D facial model and a facial model.   
     
     
         10 . The method of  claim 8 , wherein the step of projecting the average 3D facial model onto the expressionless facial model frame are replaced by 3D facial scanning. 
     
     
         11 . The method of  claim 8 , wherein the step of calculating, the step of judging, the step of producing the animation, and the step of transforming the mesh are repeated until the error is located within the minimum error range. 
     
     
         12 . The method of  claim 7 , wherein the step of transferring the 3D facial model template having the animation-controlled model onto the 3D facial model comprises transferring the 3D facial model template having the animation-controlled model onto the 3D facial model using correspondence relations between two meshes of features of a face, the features not moving with respect to facial expression change. 
     
     
         13 . The method of  claim 8 , wherein the step of transferring the 3D facial model template having the animation-controlled model onto the 3D facial model comprises transferring the 3D facial model template having the animation-controlled model onto the 3D facial model using correspondence relations between two meshes of features of a face, the features not moving with respect to facial expression change.