Neural network architecture for face tracking

Abstract

Techniques for face tracking comprise receiving landmark data associated with a plurality of images indicative of at least one facial part. Representative images corresponding to the plurality of images may be generated based on the landmark data. Each representative image may depict a plurality of segments, and each segment may correspond to a region of the at least one facial part. The plurality of images and corresponding representative images may be input into a neural network to train the neural network to predict a feature associated with a subsequently received image comprising a face. An animation associated with a facial expression may be controlled based on output from the trained neural network.

What is claimed is: 
     
       1. A method implemented by a computing system comprising at least one processor, comprising:
 receiving landmark data, wherein the landmark data are associated with a plurality of images and indicative of at least one facial part; 
 generating, based on the landmark data, representative images corresponding to the plurality of images, each representative image depicting a plurality of segments, wherein each segment corresponds to a region of the at least one facial part, wherein generating each representative image further comprises:
 rasterizing the plurality of segments with corresponding labels each of which indicates a corresponding region of the at least one facial part, and 
 delineating boundaries of the plurality of segments by applying a different color or shade to each of the plurality of segments; and 
 
 inputting the representative images corresponding to the plurality of images into a neural network and training the neural network to predict a feature associated with a subsequently received image comprising a face. 
 
     
     
       2. The method of  claim 1 , wherein the neural network is trained using the plurality of images and the representative images corresponding to the plurality of images. 
     
     
       3. The method of  claim 1 , wherein the training the neural network to predict a feature further comprises training the neural network to predict a facial expression associated with the subsequently received image comprising the face. 
     
     
       4. The method of  claim 1 , wherein the receiving landmark data further comprises:
 receiving, for each image among the plurality of images, data indicative of a plurality of numbers and a position associated with each number selected from the plurality of numbers, wherein each number selected from the plurality of numbers corresponds to a single landmark. 
 
     
     
       5. The method of  claim 4 , wherein the position associated with each number selected from the plurality of numbers comprises a set of coordinates on a corresponding image among the plurality of images. 
     
     
       6. The method of  claim 4 , further comprising:
 generating, based on the data indicative of the plurality of numbers and the position associated with each number selected from the plurality of numbers, a triangulation associated with at least one portion of the landmark data. 
 
     
     
       7. The method of  claim 6 , further comprising:
 determining, for each representative image, a boundary associated with each segment selected from the plurality of segments based on the triangulation. 
 
     
     
       8. The method of  claim 1 , wherein the region of the at least one facial part comprises one of a left eye, a right eye, a left pupil, a right pupil, a left eyebrow, a right eyebrow, a nose, an upper lip, a lower lip, or a remaining portion of the at least one facial part. 
     
     
       9. The method of  claim 1 , further comprising:
 controlling an animation associated with a facial expression based on output from the trained neural network. 
 
     
     
       10. A system, comprising:
 at least one processor in communication with at least one memory, the at least one memory comprising computer-readable instructions that upon execution by the at least one processor cause the system to perform operations comprising: 
 receiving landmark data, wherein the landmark data are associated with a plurality of images and indicative of at least one facial part; 
 generating, based on the landmark data, representative images corresponding to the plurality of images, each representative image depicting a plurality of segments, wherein each segment corresponds to a region of the at least one facial part, wherein generating each representative image further comprises:
 rasterizing the plurality of segments with corresponding labels each of which indicates a corresponding region of the at least one facial part, and 
 delineating boundaries of the plurality of segments by applying a different color or shade to each of the plurality of segments; and 
 
 inputting the representative images corresponding to the plurality of images into a neural network and training the neural network to predict a feature associated with a subsequently received image comprising a face. 
 
     
     
       11. The system of  claim 10 , wherein the neural network is trained using the plurality of images and the representative images corresponding to the plurality of images. 
     
     
       12. The system of  claim 10 , wherein the training the neural network to predict a feature further comprises training the neural network to predict a facial expression associated with the subsequently received image comprising the face. 
     
     
       13. The system of  claim 10 , wherein the receiving landmark data further comprises:
 receiving, for each image among the plurality of images, data indicative of a plurality of numbers and a position associated with each number selected from the plurality of numbers, wherein each number selected from the plurality of numbers corresponds to a single landmark. 
 
     
     
       14. The system of  claim 13 , wherein the position associated each number selected from the plurality of numbers comprises a set of coordinates on a corresponding image among the plurality of images. 
     
     
       15. The system of  claim 13 , the operations further comprising:
 generating, based on the data indicative of the plurality of numbers and the position associated with each number selected from the plurality of numbers, a triangulation associated with at least one portion of the landmark data; and 
 determining, for each representative image, a boundary associated with each segment selected from the plurality of segments based on the triangulation. 
 
     
     
       16. The system of  claim 10 , further comprising:
 controlling an amination associated with a facial expression based on output from the trained neural network. 
 
     
     
       17. A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations comprising:
 receiving landmark data, wherein the landmark data are associated with a plurality of images and indicative of at least one facial part; 
 generating, based on the landmark data, representative images corresponding to the plurality of images, each representative image depicting a plurality of segments, wherein each segment corresponds to a region of the at least one facial part, wherein generating each representative image further comprises:
 rasterizing the plurality of segments with corresponding labels each of which indicates a corresponding region of the at least one facial part, and 
 delineating boundaries of the plurality of segments by applying a different color or shade to each of the plurality of segments; and 
 
 inputting the representative images corresponding to the plurality of images into a neural network and training the neural network to predict a feature associated with a subsequently received image comprising a face. 
 
     
     
       18. The non-transitory computer-readable storage medium of  claim 17 , wherein the neural network is trained using the plurality of images and the representative images corresponding to the plurality of images. 
     
     
       19. The non-transitory computer-readable storage medium of  claim 17 , the operations further comprising:
 receiving, for each image among the plurality of images, data indicative of a plurality of numbers and a position associated with each number selected from the plurality of numbers, wherein each number selected from the plurality of numbers corresponds to a single landmark; 
 generating, based on the data indicative of the plurality of numbers and the position associated with each number selected from the plurality of numbers, a triangulation associated with at least one portion of the landmark data; and 
 determining, for each representative image, a boundary associated with each segment selected from the plurality of segments based on the triangulation.