Synthetic video generation showing modified face

Abstract

A system comprises an image capture device, a processing device, and a display. The image capture device captures a stream of images of a face of a person showing a current dentition of the person and soft facial tissues of the person. The processing device processes the stream of images to generate a modified stream of images showing a modified dentition of the person and modified soft facial tissues of the person associated with the modified dentition. The display displays the modified stream of images.

What is claimed is: 
     
       1. A system comprising:
 an image capture device to capture a stream of images of a face of a person showing a current dentition of the person and soft facial tissues of the person; 
 a processing device to process the stream of images to generate a modified stream of images showing a modified dentition of the person and modified soft facial tissues of the person associated with the modified dentition, wherein to generate a modified version of an image of the stream of images the processing device is to:
 identify a first plurality of landmarks associated with the current dentition of the person and a second plurality of landmarks associated with the soft facial tissues of the person from the image; 
 determine a modified arrangement of the first plurality of landmarks based on a first mapping between the current dentition of the person and the modified dentition of the person; and 
 determine, based on applying the modified arrangement of the first plurality of landmarks to a second mapping between the first plurality of landmarks and the second plurality of landmarks, a modified shape of the soft facial tissues; and 
 
 a display to display the modified stream of images. 
 
     
     
       2. The system of  claim 1 , wherein the display is to display one or more modified images of the modified stream of images while the image capture device captures one or more images of the stream of images. 
     
     
       3. The system of  claim 1 , wherein the image capture device, the processing device and the display are components of a mobile computing device. 
     
     
       4. The system of  claim 1 , wherein the processing device is further to:
 output instructions for the person to adopt a series of facial expressions, wherein the stream of images comprise images associated with the series of facial expressions. 
 
     
     
       5. The system of  claim 4 , wherein the processing device is further to:
 make a determination of how the soft facial tissues respond to the series of facial expressions from the images associated with the series of facial expressions; 
 wherein the modified shape of the soft facial tissues is determined at least in part based on the determination of how the soft facial tissues respond to the series of facial expressions. 
 
     
     
       6. The system of  claim 1 , wherein the first mapping and the second mapping are components of a facial model that generates post-treatment images comprising the modified dentition and the soft facial tissues based on pre-treatment images comprising the current dentition and the soft facial tissues. 
     
     
       7. The system of  claim 1 , wherein the stream of images comprises a stream of three-dimensional (3D) images. 
     
     
       8. The system of  claim 7 , wherein the modified stream of images comprises a stream of 3D images, and wherein displaying the modified stream of images comprises displaying a 2D projection of the modified stream of images. 
     
     
       9. The system of  claim 1 , wherein the processing device is further to:
 register the stream of images to a virtual 3D model of an upper dental arch and a lower dental arch of the person. 
 
     
     
       10. The system of  claim 1 , wherein the processing device is further to:
 receive a user selection of a post-treatment dentition of the person; and 
 determine the first mapping based at least in part on the post-treatment dentition of the person. 
 
     
     
       11. A method comprising:
 receiving a stream of images of a face of a person showing a current dentition of the person and soft facial tissues of the person; 
 for each image of the stream of images, generating a modified version of the image showing a modified dentition of the person and modified soft facial tissues of the person associated with the modified dentition, wherein generating a modified version of an image of the stream of images comprises:
 identifying a first plurality of landmarks and a second plurality of landmarks on the face from the image, wherein the second plurality of landmarks are associated with soft facial tissues; 
 determining a post-treatment arrangement of the first plurality of landmarks based on a first mapping between a current dentition of the person and a post-treatment dentition of the person; and 
 determining, based on applying the post-treatment arrangement of the first plurality of landmarks to a second mapping between the first plurality of landmarks and the second plurality of landmarks, a post-treatment shape of the soft facial tissues; and 
 outputting a stream of modified images of the face of the person. 
 
 
     
     
       12. The method of  claim 11 , wherein the first mapping and the second mapping are components of a facial model that generates post-treatment images comprising the modified dentition and the soft facial tissues based on pre-treatment images comprising the current dentition and the soft facial tissues. 
     
     
       13. The method of  claim 11 , wherein the stream of modified images is a stream of three-dimensional (3D) images, the method further comprising:
 generating two-dimensional (2D) projections of the stream of modified images; and 
 outputting the 2D projections to a display. 
 
     
     
       14. The method of  claim 11 , further comprising:
 registering the stream of images to a virtual 3D model of an upper dental arch and a lower dental arch of the person. 
 
     
     
       15. The method of  claim 11 , further comprising:
 displaying one or more modified images of the stream of modified images while an image capture device captures one or more images of the stream of images. 
 
     
     
       16. The method of  claim 11 , wherein:
 a) the stream of images is received from a mobile computing device; or 
 b) the stream of images is received by a processing device of a mobile computing device from one or more other components of the mobile computing device. 
 
     
     
       17. The method of  claim 11 , further comprising:
 outputting instructions for the person to perform a plurality of facial expressions, wherein the stream of images comprise images associated with the plurality of facial expressions. 
 
     
     
       18. The method of  claim 17 , further comprising:
 making a determination of how the soft facial tissues respond to the plurality of facial expressions from the images associated with the plurality of facial expressions; 
 wherein the post-treatment shape of the soft facial tissues is determined at least in part based on the determination of how the soft facial tissues respond to the plurality of facial expressions. 
 
     
     
       19. A system comprising:
 a computing device comprising a processor and a memory, the computing device to:
 receive a stream of images of a face of a person showing a current dentition of the person and soft facial tissues of the person; 
 process the stream of images to generate a modified stream of images showing a modified dentition of the person and modified soft facial tissues of the person associated with the modified dentition, wherein to generate a modified version of an image of the stream of images the computing device is to:
 identify a first plurality of landmarks associated with the current dentition of the person and a second plurality of landmarks associated with the soft facial tissues of the person from the image; 
 determine a modified arrangement of the first plurality of landmarks based on a first mapping between the current dentition of the person and the modified dentition of the person; and 
 determine, based on applying the modified arrangement of the first plurality of landmarks to a second mapping between the first plurality of landmarks and the second plurality of landmarks, a modified shape of the soft facial tissues; and 
 
 transmit the modified stream of images to a remote device. 
 
 
     
     
       20. The system of  claim 19 , further comprising:
 the remote device, wherein the remote device is to receive the modified stream of images and output the modified stream of images to a display.