Shape prediction model compression for face alignment

Abstract

A face tracking system generates a model for extracting a set of facial anchor points on a face within a portion of a face image based a multiple-level cascade of decision trees. The face tracking system identifies a mesh shape adjusted to an image of a face. For each decision tree, the face tracking system identifies an adjustment vector for the mesh shape relative to the image of the face. For each cascade level, the face tracking system combines the identified adjustment for each decision tree to determine a combined adjustment vector for the cascade level. The face tracking system modifies adjustment of the mesh shape to the face in the image based on the combined adjustment vector. The face tracking system reduces the model to a dictionary and atom weights using a learned dictionary. The model may be more easily transmitted to devices and stored on devices.

What is claimed is: 
     
       1. A computer-implemented method comprising:
 training, by a computer using a set of sample facial images, a full-size model for adjusting a mesh shape to match facial anchor points to an image of a face, the trained full-size model having a set of decision trees, each decision tree specifying a plurality of leaf nodes accessible by logic for traversing the decision tree, each leaf node specifying a learned set of adjustment values upon which adjustments of the facial anchor points in the mesh shape are determined, the adjustment values learned in the training using the set of sample facial images, wherein the leaf nodes of the trained full-size model in total have a first number of values; 
 for each leaf node of one or more leaf nodes of the trained full-size model:
 associating a set of one or more atoms with the leaf node, each atom describing a function to adjust one or more of the adjustment values, the associating comprising:
 (i) identifying one or more atoms from a dictionary of atoms, 
 (ii) determining that a candidate weighted combination of the atoms generates a second set of adjustment values that is within a threshold level of approximation of the learned set of adjustment values specified by the leaf node, and 
 (iii) assigning weights to the atoms to the leaf node based on the candidate weighted combination of the atoms; 
 
 
 generating, from the full-size model, a reduced-size model that includes the dictionary of atoms, a reduced set of trees including the logic for traversing the set of trees to leaf nodes, and the weights of the set of one or more atoms associated with each leaf node, wherein the leaf nodes in the reduced-size model in total have a second number of values smaller than the first number; and 
 transmitting the reduced-size model to a client mobile device having an imaging device capable of capturing facial images, wherein, in applying the reduced-size model to a captured facial image, the second set of adjustment values of a given leaf node is generated by a processor of the client mobile device by applying the weights to the set of one or more atoms associated with the given leaf node. 
 
     
     
       2. The method of  claim 1 , wherein applying the reduced-size model to the captured facial image comprises:
 traversing a tree of the reduced set of trees in the reduced set according to the logic to identify one of the leaf nodes; 
 generating the second set of adjustment values by combining the set of one or more atoms for the one of the leaf nodes and the weights of the atoms; and 
 adjusting the mesh shape based on the second set of adjustment values. 
 
     
     
       3. The method of  claim 2 , wherein the reduced-size model is compressed, and the reduced-size model is partially decompressed for atoms associated with the identified one of the leaf nodes. 
     
     
       4. The method of  claim 2 , wherein the atoms that are not associated with the identified one of the leaf nodes are not accessed in generating the second set of adjustment values. 
     
     
       5. The method of  claim 1 , wherein the set of one or more atoms and corresponding weights for each of the leaf nodes are determined based on a matching pursuit algorithm. 
     
     
       6. The method of  claim 5 , wherein the matching pursuit algorithm is an orthogonal matching pursuit algorithm. 
     
     
       7. The method of  claim 1 , wherein the learned set of adjustment values specified by each of the leaf nodes is represented as a sparse vector. 
     
     
       8. A method comprising:
 receiving, by a client mobile device having an image device capable of capturing facial images, a reduced-size model that comprises (i) a dictionary of atoms each describing a function for adjusting a mesh shape, and (ii) a reduced set of trees including logic for traversing the reduced set of trees to reduced-size leaf nodes, the reduced-size model generated from a full-size model having a plurality of full-size leaf nodes each specifying a first set of adjustment values, the full-size model in total having a first number of values and the reduced-size model in total having a second number of values smaller than the first number, wherein the reduced-size model is generated by:
 for each leaf node of one or more full-size leaf nodes of the full-size model:
 associating a set of one or more atoms with the leaf node, the associating comprising:
 (i) identifying one or more atoms from the dictionary of atoms, 
 (ii) determining that a candidate weighted combination of the atoms generates a second set of adjustment values that is within a threshold level of approximation of the first set of adjustment values specified by the full-size leaf node, and 
 (iii) assigning weights to the atoms to the leaf node based on the candidate weighted combination of the atoms; 
 
 
 
 capturing a facial image by the image device; and 
 matching a current mesh shape with facial anchor points of the captured facial image, the matching comprising:
 traversing one of the trees according to the logic and the facial anchor points to identify a traversed reduced-size leaf node that is associated with a set of one or more atoms; 
 generating the second set of adjustment values associated with the identified traversed leaf node by combining the set of one or more atoms for the identified traversed leaf node and the weights of the set of one or more atoms; and 
 adjusting the current mesh shape based on the generated second set of adjustment values. 
 
 
     
     
       9. The method of  claim 8 , wherein the reduced-size model is compressed, and the reduced-size model is partially decompressed for atoms associated with identified traversed leaf nodes. 
     
     
       10. The method of  claim 8 , wherein the atoms that are not identified by identified traversed leaf nodes are not accessed in generating the second set of adjustment values. 
     
     
       11. The method of  claim 8 , wherein the second set of adjustment values is not stored as a part of the reduced-size model. 
     
     
       12. A non-transitory computer-readable medium having instructions that when executed by a processor perform the steps of:
 training, by a computer using a set of sample facial images, a full-size model for adjusting a mesh shape to match facial anchor points to an image of a face, the trained full-size model having a set of decision trees, each decision tree specifying a plurality of leaf nodes accessible by logic for traversing the decision tree, each leaf node specifying a learned set of adjustment values upon which adjustments of the facial anchor points in the mesh shape are determined, the adjustment values learned in the training using the set of sample facial images, wherein the leaf nodes of the trained full-size model in total have a first number of values; 
 for each leaf node of one or more leaf nodes of the trained full-size model:
 associating a set of one or more atoms with the leaf node, each atom describing a function to adjust one or more of the adjustment values, the associating comprising:
 (i) identifying one or more atoms from a dictionary of atoms, 
 (ii) determining that a candidate weighted combination of the atoms generates a second set of adjustment values that is within a threshold level of approximation of the learned set of adjustment values specified by the leaf node, and 
 (iii) assigning weights to the atoms to the leaf node based on the candidate weighted combination of the atoms; 
 
 
 generating, from the full-size model, a reduced-size model that includes the dictionary of atoms, a reduced set of trees including the logic for traversing the set of trees to leaf nodes, and the weights of the set of one or more atoms associated with each leaf node, wherein the leaf nodes in the reduced-size model in total have a second number of values smaller than the first number; and 
 transmitting the reduced-size model to a client mobile device having an imaging device capable of capturing facial images, wherein, in applying the reduced-size model to a captured facial image, the second set of adjustment values of a given leaf node is generated by a processor of the client mobile device by applying the weights to the set of one or more atoms associated with the given leaf node. 
 
     
     
       13. The non-transitory computer-readable medium of  claim 12 , wherein applying the reduced-size model to the captured facial image comprises:
 traversing a tree of the reduced set of trees in the reduced set according to the logic to identify one of the leaf nodes; 
 generating the second set of adjustment values by combining the set of one or more atoms for the one of the leaf nodes and the weights of the atoms; and 
 adjusting the mesh shape based on the second set of adjustment values. 
 
     
     
       14. The non-transitory computer-readable medium of  claim 13 , wherein the reduced-size model is compressed, and the reduced-size model is partially decompressed for atoms associated with the identified one of the leaf nodes. 
     
     
       15. The non-transitory computer-readable medium of  claim 13 , wherein the atoms that are not associated with the identified one of the leaf nodes are not accessed in generating the second set of adjustment values. 
     
     
       16. The non-transitory computer-readable medium of  claim 12 , wherein the set of one or more atoms and corresponding weights for each of the leaf nodes are determined based on a matching pursuit algorithm. 
     
     
       17. The non-transitory computer-readable medium of  claim 16 , wherein the matching pursuit algorithm is an orthogonal matching pursuit algorithm. 
     
     
       18. The non-transitory computer-readable medium of  claim 12 , wherein the learned set of adjustment values specified by each of the leaf nodes is represented as a sparse vector. 
     
     
       19. A non-transitory computer-readable medium having instructions that when executed by a processor perform the steps of:
 receiving, by a client mobile device having an image device capable of capturing facial images, a reduced-size model that comprises (i) a dictionary of atoms each describing a function for adjusting a mesh shape, and (ii) a reduced set of trees including logic for traversing the reduced set of trees to reduced-size leaf nodes, the reduced-size model generated from a full-size model having a plurality of full-size leaf nodes each specifying a first set of adjustment values, the full-size model in total having a first number of values and the reduced-size model in total having a second number of values smaller than the first number, wherein the reduced-size model is generated by:
 for each leaf node of one or more full-size leaf nodes of the full-size model:
 associating a set of one or more atoms with the leaf node, the associating comprising:
 (i) identifying one or more atoms from the dictionary of atoms, 
 (ii) determining that a candidate weighted combination of the atoms generates a second set of adjustment values that is within a threshold level of approximation of the first set of adjustment values specified by the full-size leaf node, and 
 (iii) assigning weights to the atoms to the leaf node based on the candidate weighted combination of the atoms; 
 
 
 
 capturing a facial image by the image device; and 
 matching a current mesh shape with facial anchor points of the captured facial image, the matching comprising:
 traversing one of the trees according to the logic and the facial anchor points to identify a traversed reduced-size leaf node that is associated with a set of one or more atoms; 
 generating the second set of adjustment values associated with the identified traversed leaf node by combining the set of one or more atoms for the identified traversed leaf node and the weights of the set of one or more atoms; and 
 adjusting the current mesh shape based on the generated second set of adjustment values. 
 
 
     
     
       20. The non-transitory computer-readable medium of  claim 19 , wherein the reduced-size model is compressed, and the reduced-size model is partially decompressed for atoms associated with identified traversed leaf nodes. 
     
     
       21. The non-transitory computer-readable medium of  claim 19 , wherein the atoms that are not identified by identified traversed leaf nodes are not accessed in generating the second set of adjustment values. 
     
     
       22. The non-transitory computer-readable medium of  claim 19 , wherein the second set of adjustment values is not stored as a part of the reduced-size model.