Method and system for real-time facial image enhancement

Abstract

The present invention is a system and method for detecting facial features of humans in a continuous video and super-imposing virtual objects onto the features automatically and dynamically in real-time. The suggested system is named Facial Enhancement Technology (FET). The FET system consists of three major modules, initialization module, facial feature detection module, and superimposition module. Each module requires demanding processing time and resources by nature, but the FET system integrates these modules in such a way that real time processing is possible. The users can interact with the system and select the objects on the screen. The superimposed image moves along with the user's random motion dynamically. The FET system enables the user to experience something that was not possible before by augmenting the person's facial images. The hardware of the FET system comprises the continuous image-capturing device, image processing and controlling system, and output display system.

1. A method for enhancing facial images of a person in a video by superimposing virtual object images onto said facial images automatically and dynamically in real-time, comprising the following steps of :
 (a) capturing a plurality of images for said person with at least one means for capturing images, 
 (b)  processing said  a plurality of images of said person in order to detect the  a face of said person in said plurality of images for an initial face detection,  
 (c)  tracking the detected face in real-time using a face tracking window,  
 (d)  verifying said face within the boundary of the face tracking window using a second local face detection,  
 (e)  calculating facial feature coordinate information from said face by applying a plurality of facial feature detection approaches and applying a block-processing,  
 (f)  processing a touch-free interaction by said person to select virtual object images, and  
 (g)  superimposing said virtual object images onto said facial images automatically and dynamically in real-time, wherein the virtual object images comprise predefined virtual objects that can be  are attached able to and enhance  the facial images,  
 whereby the verification ofverifying said face within the boundary of the face tracking window makes more accurate facial images to be passed on to the next step  while enabling real-time tracking of the face, and  
 whereby saidenabling said person to choose said virtual object images through touch-free interaction enables said person to choose said virtual object images  on a means for displaying output to superimpose onto said facial images without touching any device .  
 
     
     
       2. The method according to  claim 1 , wherein the method further comprises a step of  repeating the steps from (a) to (g)  method for a plurality of persons. 
     
     
       3. The method according to  claim 1 , wherein the method further comprises a step of :
 dynamically estimating regions of interest for eachat least one facial feature in said face dynamically,  
 wherebycausing said regions of interest to change according to the results from the step of  verifying said face within the boundary of the face tracking window, and  
 wherebyusing said regions of interest are used  as boundaries for detecting each  the at least one facial feature, such as eyes, nose, and mouth  on said face.  
 
     
     
       4. The method according to  claim 1 , wherein the method further comprises a step of  applying fusion algorithms and geometrical constraints to said facial feature coordinate information. 
     
     
       5. The method according to  claim 1 , wherein the method further comprises a step of  smoothing said facial feature coordinate information. 
     
     
       6. The method according to  claim 5 , wherein the method further comprises a step of  storing a history of a plurality of said facial feature coordinate information and applying a smoothing algorithm for the current facial feature coordinate using said history of said facial feature coordinate information. 
     
     
       7. The method according to  claim 1 , wherein the method further comprises a step of  combining the face detection process and the real-time face tracking process,
 whereby the integration enables efficient and robust real-time facial image processing .  
 
     
     
       8. The method according to  claim 1 , wherein the method further comprises a step of  applying said block-processing to each  at least a subset of said plurality of facial feature detection approaches. 
     
     
       9. The method according to  claim 1 , wherein the step of (g)  superimposing said virtual object images onto said facial images automatically and dynamically in real-time further comprises steps of :
 (a)  preparing virtual object images,  
 (b)  validating said facial feature coordinate information,  
 (c)  smoothing,  
 (d)  aesthetic processing, and  
 (e)  processing final superimposition.  
 
     
     
       10. The method according to  claim 9 , wherein the step of (a)  preparing virtual object images further comprises a step for  setting pivot points in said virtual object images. 
     
     
       11. The method according to  claim 9 , wherein the step of (d)  aesthetic processing further comprises a step for  processing rotation and translation of said virtual object images. 
     
     
       12. An apparatus for enhancing  configured to enhance facial images of a person in a video by superimposing virtual object images onto said facial images automatically and dynamically in real-time, comprising the following steps of  the apparatus comprising:
 (a) capturing a plurality of images for said person with at least one means for capturing images, 
 (b)means for processing saida plurality of images of said person in order to detect thea face of said person in said plurality of images for an initial face detection,  
 (c)means for tracking the detected face in real-time using a face tracking window,  
 (d)means for verifying said face within the boundary of the face tracking window using a second local face detection,  
 (e)means for calculating facial feature coordinate information from said face by applying a plurality of facial feature detection approaches and applying a block-processing,  
 (f)means for processing a touch-free interaction by said person to select virtual object images, and  
 (g)means for superimposing said virtual object images onto said facial images automatically and dynamically in real-time,  
 wherein the virtual object images comprise predefined virtual objects which  that can be  are attached able to and enhance  the facial images,  
 whereby the verification ofwherein said means for verifying said face within the boundary of the face tracking window makes more accurateis configured to increase accuracy of the facial images to be passed on to the next step  while enabling real-time tracking of the face, and  
 whereby saidwherein said means for processing the touch-free interaction is configured to enables  said person to choose said virtual object images on a means for displaying output to superimpose onto said facial images without touching any device .  
 
     
     
       13. The apparatus according to  claim 12 , wherein the apparatus further comprises means for repeatedly using the means from (a) to (g)  apparatus to enhance facial images for a plurality of persons. 
     
     
       14. The apparatus according to  claim 12 , wherein the apparatus further comprises means for estimating regions of interest for each  at least one facial feature in said face dynamically,
 whereby said regions of interest change according to the results from using the means for verifying said face within the boundary of the face tracking window, and  
 whereby said regions of interest are used as boundaries for detecting each  the at least one facial feature, such as eyes, nose, and mouth  on said face.  
 
     
     
       15. The apparatus according to  claim 12 , wherein the apparatus further comprises means for applying fusion algorithms and geometrical constraints to said facial feature coordinate information. 
     
     
       16. The apparatus according to  claim 12 , wherein the apparatus further comprises means for smoothing said facial feature coordinate information. 
     
     
       17. The apparatus according to  claim 16 , wherein the apparatus further comprises means for storing a history of a plurality of said facial feature coordinate information and applying a smoothing algorithm for the current facial feature coordinate using said history of said facial feature coordinate information. 
     
     
       18. The apparatus according to  claim 12 , wherein the apparatus further comprises means for combining the face detection process and the real-time face tracking process,
 whereby the integration enables efficient and robust real-time facial image processing .  
 
     
     
       19. The apparatus according to  claim 12 , wherein the apparatus further comprises means for applying said block-processing to each  at least one of said plurality of facial feature detection approaches. 
     
     
       20. The apparatus according to  claim 12 , wherein the means for (g)  superimposing said virtual object images onto said facial images automatically and dynamically in real-time further comprises means for :
 (a)means for preparing virtual object images,  
 (b)means for validating said facial feature coordinate information,  
 (c)means for smoothing,  
 (d)means for aesthetic processing, and  
 (e)means for processing final superimposition.  
 
     
     
       21. The apparatus according to  claim 20 , wherein the means for (a)  preparing virtual object images further comprises means for setting pivot points in said virtual object images. 
     
     
       22. The apparatus according to  claim 20 , wherein the means for (d)  aesthetic processing further comprises means for processing rotation and translation of said virtual object images. 
     
     
       23. A system configured to enhance facial images of a person in a video by superimposing virtual object images onto said facial images automatically and dynamically in real- time, the system comprising:      program instructions stored in data storage, wherein the program instructions, in response to execution by a computing device, cause the computing device to:      detect a face of said person in a plurality of images for an initial face detection;        track the detected face in real - time using a face tracking window;        verify said face within the boundary of the face tracking window using a second local face detection, whereby the verification of said face within the boundary of the face tracking window increases accuracy of the facial images while enabling real - time tracking of the face;        calculate facial feature coordinate information from said face by applying a plurality of facial feature detection approaches and applying a block - processing, wherein the virtual object images comprise predefined virtual objects which that are attachable to the facial images;        process a touch - free interaction by said person to select virtual object images, wherein said touch - free interaction enables said person to choose said virtual object images on a means for displaying output to superimpose onto said facial images; and        superimpose said virtual object images onto said facial images automatically and dynamically in real - time.       
     
     
       24. The system according to  claim 23 , wherein the program instructions, in response to execution by the computing device, further cause the computing device to estimate dynamically at least one region of interest for at least one facial feature in said face,
   wherein said regions of interest change according to the results of verifying said face within the boundary of the face tracking window, and        wherein said regions of interest are used as boundaries for detecting at least one facial feature on said face.     
     
     
       25. The system according to  claim 23 , wherein the program instructions, in response to execution by the computing device, further cause the computing device to apply fusion algorithms and geometrical constraints to said facial feature coordinate information. 
     
     
       26. The system according to  claim 23 , wherein the program instructions, in response to execution by the computing device, further cause the computing device to smooth said facial feature coordinate information. 
     
     
       27. The system according to  claim 26 , wherein the program instructions, in response to execution by the computing device, further cause the computing device to store a history of a plurality of said facial feature coordinate information and apply a smoothing algorithm for the current facial feature coordinate using said history of said facial feature coordinate information. 
     
     
       28. The system according to  claim 23 , wherein the program instructions, in response to execution by the computing device, further cause the computing device to combine the face detection process and the real- time face tracking process.   
     
     
       29. The system according to  claim 23 , wherein the program instructions, in response to execution by the computing device, further cause the computing device to apply said block- processing to at least one of said plurality of facial feature detection approaches.   
     
     
       30. A non- transitory tangible computer readable medium having instructions stored thereon, the instructions comprising:      instructions for processing a plurality of images of said person in order to detect a face of said person in said plurality of images for an initial face detection,        instructions for tracking the detected face in real - time using a face tracking window,        instructions for verifying said face within the boundary of the face tracking window using a second local face detection,        instructions for calculating facial feature coordinate information from said face by applying a plurality of facial feature detection approaches and applying a block - processing,        instructions for processing a touch - free interaction by said person to select virtual object images, and        instructions for superimposing said virtual object images onto said facial images automatically and dynamically in real - time, wherein the virtual object images comprise predefined virtual objects that are attachable to the facial images,        instructions for verifying said face within the boundary of the face tracking window while enabling real - time tracking of the face, and        instructions for enabling said person to chose said virtual object images through touch - free interaction on a means for displaying output to superimpose onto said facial images.     
     
     
       31. The non- transitory tangible computer readable medium according to    claim 30   , further comprising:      instructions for estimating regions of interest for at least one facial feature in said face dynamically,        instructions for causing said regions of interest to change according to the results from verifying said face within the boundary of the face tracking window, and        instructions for using said regions of interest as boundaries for detecting at least one facial feature on said face.     
     
     
       32. The non- transitory tangible computer readable medium according to    claim 30   , wherein the instructions stored on the non - transitory tangible computer readable medium further comprise:      instructions for applying fusion algorithms and geometrical constraints to said facial feature coordinate information.     
     
     
       33. The non- transitory tangible computer readable medium according to    claim 30   , further comprising:      instructions for smoothing said facial feature coordinate information.     
     
     
       34. The non- transitory tangible computer readable medium according to    claim 33   , further comprising:      instructions for storing a history of a plurality of said facial feature coordinate information, and        instructions for applying a smoothing function for the current facial feature coordinate using said history of said facial feature coordinate information.     
     
     
       35. The non- transitory tangible computer readable medium according to    claim 30   , further comprising:      instructions for combining the face detection process and the real - time face tracking process.     
     
     
       36. The non- transitory tangible computer readable medium according to    claim 30   , further comprising:      instructions for applying said block - processing to at least one of said plurality of facial feature detection approaches.