Method and system for the automatic computerized audio visual dubbing of movies

Abstract

A method for automated computerized audio visual dubbing of a movie comprises (i) generating a three-dimensional head model of an actor in a movie, the head model being representative of specific facial features of the actor, and (ii) generating a three-dimensional head model of a dubber making target sounds for the actor, the dubber head model being representative of specific facial features of the dubber, as the target sounds are made. The method also comprises modifying at least a portion of the specific facial features of the actor head model according to the dubber head model such that the actor appears to be producing target sounds made by the dubber.

1 . A computer readable medium comprising instructions for causing a computer to carry out a method for automated computerized audio visual dubbing of a movie, said method comprising: 
 (a) generating a three-dimensional head model of an actor in a movie for at least one frame in said movie, wherein said actor head model is representative of specific facial features of said actor in said frame;    (b) generating a three-dimensional head model of a dubber making target sounds for said at least one frame in said movie, wherein said dubber head model is representative of specific facial features of said dubber, as said target sounds are made; and    (c) modifying at least a portion of said specific facial features of said actor head model according to said dubber head model such that said actor appears to be producing said target sounds made by said dubber.    
     
     
         2 . A computer-readable medium according to  claim 1 , said method further comprising the step of modifying the face of the actor in said frame according to said actor head model as modified in step (c) so as to obtain a frame wherein at least a portion of the specific facial features of the actor correspond to specific facial features of said dubber and replacing target sounds made by said actor with target sounds made by said dubber.  
     
     
         3 . A computer-readable medium according to  claim 1 , wherein said specific facial features comprise the lips and mouth area.  
     
     
         4 . A computer-readable medium according to  claim 1 , wherein said at least one frame comprises multiple frames of a movie.  
     
     
         5 . A computer-readable medium according to  claim 1 , wherein said movie comprises a photographed movie.  
     
     
         6 . A computer-readable medium according to  claim 1 , wherein said movie comprises an animated movie.  
     
     
         7 . A computer-readable medium according to  claim 1 , wherein steps (a) and (b) of generating the three-dimensional head models are accomplished by computer fitting a generic three-dimensional head model to the actor or dubber's picture using significant facial features of said actor or dubber.  
     
     
         8 . A computer-readable medium according to  claim 7 , said method further comprising tracking specific facial features of said three-dimensional head model of said actor or said dubber through a plurality of frames in said movie so as to create a library of reference similarity frames.  
     
     
         9 . A computer-readable medium according to  claim 8 , said method further comprising mapping, for a plurality of frames in the movie, the face of the actor to the three dimensional head model of said actor wherein said mapping employs a computerized texture mapping technique that uses said reference similarity frames.  
     
     
         10 . A computer-readable medium according to  claim 1 , wherein the step of modifying at least a portion of said three-dimensional head model of said actor comprises replacing, on a frame by frame basis, at least a portion of specific facial features of the actor head model with those of the dubber head model.  
     
     
         11 . A method for audio visual dubbing of a movie, comprising: 
 (a) generating a first model of an actor in a movie for at least one frame in said movie;    (b) generating a second model of a dubber making target sounds for said at least one frame in said movie; and    (c) modifying at least a portion of said first model according to said second model such that said actor appears to be producing said target sounds made by said dubber.    
     
     
         12 . A method according to  claim 11 , wherein said first model comprises a three-dimensional head model representative of specific facial features of said actor in said frame.  
     
     
         13 . A method according to  claim 11 , wherein said second model comprises a three-dimensional head model representative of specific facial features of said dubber, as said target sounds are made.  
     
     
         14 . A method according to  claim 11 , further comprising modifying the face of the actor in said frame according to said first model as modified in step (c) so as to obtain a frame wherein at least a portion of specific facial features of the actor correspond to specific facial features of said dubber and replacing target sounds made by said actor with target sounds made by said dubber.  
     
     
         15 . A method according to  claim 14 , wherein said specific facial features comprise the lips and mouth area.  
     
     
         16 . A method according to  claim 11 , wherein said at least one frame comprises multiple frames of a movie.  
     
     
         17 . A method according to  claim 11 , wherein said movie comprises a photographed movie.  
     
     
         18 . A method according to  claim 11 , wherein said movie comprises an animated movie.  
     
     
         19 . A method according to  claim 11 , wherein steps (a) and (b) are accomplished by computer fitting a generic three-dimensional head model to the actor or dubber's picture using significant facial features of said actor or dubber.  
     
     
         20 . A method according to  claim 11 , further comprising tracking specific facial features of said first model through a plurality of frames in said movie so as to create a library of reference similarity frames.  
     
     
         21 . A method according to  claim 11 , further comprising tracking specific facial features of said second model through a plurality of frames in said movie so as to create a library of reference similarity frames.  
     
     
         22 . A method according to  claim 21 , further comprising mapping, for a plurality of frames in the movie, the face of the actor to the first model, wherein said mapping employs a computerized texture mapping technique that uses said reference similarity frames.  
     
     
         23 . A method according to  claim 11 , wherein the step of modifying at least a portion of said first model comprises replacing, on a frame by frame basis, at least a portion of specific facial features of the first model with those of the second model.  
     
     
         24 . A system for audio visual dubbing of a movie, comprising: 
 (a) means for generating a first model of an actor in a movie for at least one frame in said movie;    (b) means for generating a second model of a dubber making target sounds for said at least one frame in said movie; and    (c) means for modifying at least a portion of said first model according to said second model such that said actor appears to be producing said target sounds made by said dubber.    
     
     
         25 . A system according to  claim 24 , wherein said first model comprises a three-dimensional head model representative of specific facial features of said actor in said frame.  
     
     
         26 . A system according to  claim 24 , wherein said second model comprises a three-dimensional head model representative of specific facial features of said dubber, as said target sounds are made.  
     
     
         27 . A system according to  claim 24 , further comprising means for modifying the face of the actor in said frame according to said first model so as to obtain a frame wherein at least a portion of specific facial features of the actor correspond to specific facial features of said dubber and replacing target sounds made by said actor with target sounds made by said dubber.  
     
     
         28 . A system according to  claim 27 , wherein said specific facial features comprise the lips and mouth area.  
     
     
         29 . A system according to  claim 24 , wherein said at least one frame comprises multiple frames of a movie.  
     
     
         30 . A system according to  claim 24 , wherein said movie comprises a photographed movie.  
     
     
         31 . A system according to  claim 24 , wherein said movie comprises an animated movie.  
     
     
         32 . A system according to  claim 24 , further comprising means for computer fitting a generic three-dimensional head model to the actor or dubber's picture using significant facial features of said actor or dubber.  
     
     
         33 . A system according to  claim 24 , further comprising means for tracking specific facial features of said first model through a plurality of frames in said movie so as to create a library of reference similarity frames.  
     
     
         34 . A system according to  claim 24 , further comprising means for tracking specific facial features of said second model through a plurality of frames in said movie so as to create a library of reference similarity frames.  
     
     
         35 . A system according to  claim 34 , further comprising means for mapping, for a plurality of frames in the movie, the face of the actor to the first model, wherein said mapping employs a computerized texture mapping technique that uses said reference similarity frames.  
     
     
         36 . A system according to  claim 24 , further comprising means for replacing, on a frame by frame basis, at least a portion of specific facial features of the first model with those of the second model.