Video face recognition

Abstract

The present invention is a complete end-to-end video face recognition system. The invention performs a track-by-track labeling of all tracks within a video. A novel algorithm Mean Sequence SRC (MSSRC) is applied to a face track to perform video face recognition using a joint optimization to leverage all of the available video data and the knowledge that the face track frames belong to the same individual. Additionally the system constructs a probabilistic affinity graph combining appearance and co-occurrence similarities to model the relationship between face tracks in a video. Finally, using this relationship graph, random walk analysis is employed to propagate strong class predictions among similar face tracks, while dampening weak predictions.

What is claimed is: 
     
       1. A non-transitory computer-readable medium having stored thereon computer-executable instructions that, in response to being executed by a computing device, cause the computing device to perform operations comprising:
 receiving a video clip having a plurality of frames; 
 receiving a dictionary of still images of a plurality of faces; 
 detecting a first face within the video clip; 
 tracking the first face over the plurality of frames of the video clip to obtain a first face track; 
 computing a mean of the first face track; 
 performing an l 1 -minimization on the mean of the first face track under a predefined sparsity constraint; and 
 computing class probabilities to establish an initial identity of the first face, wherein the identity of the first face is selected from the dictionary. 
 
     
     
       2. The computer-readable medium of  claim 1 , wherein a single coefficient vector is imposed on the plurality of frames within the first face track. 
     
     
       3. The computer-readable medium of  claim 1 , wherein the dictionary is normalized to obtain a unit l 2 -norm. 
     
     
       4. The computer-readable medium of  claim 1 , further comprising calculating a confidence value, wherein the confidence value represents a likelihood of the first face being correctly identified. 
     
     
       5. The computer-readable medium of  claim 1 , further comprising:
 detecting a second face within the video clip; 
 comparing location and appearance metrics of the second face and the first face; and 
 responsive to the location and appearance metrics of the second face and the first face exhibiting a predefined level of similarity, associating the second face with the first face track. 
 
     
     
       6. The computer-readable medium of  claim 1 , further comprising:
 detecting a second face within the video clip; 
 establishing a first bounding box encompassing the first face, a second bounding box encompassing the second face, and a third bounding box encompassing both the first and the second faces; 
 calculating a ratio of the third bounding box to the first or the second bounding boxes; and 
 responsive to the ratio satisfying a predetermined value, associating the second face with the first face track. 
 
     
     
       7. The computer-readable medium of  claim 1 , further comprising obtaining a global histogram for a frame and using the global histogram to evaluate whether the frame is associated with the face track. 
     
     
       8. The computer-readable medium of  claim 1 , further comprising ending the first face track in response to the first face not being detected in a predetermined number of frames following a frame in which the first face was initially detected. 
     
     
       9. The computer-readable medium of  claim 1 , wherein local histograms are used to determine whether a second face is associated with the first face track. 
     
     
       10. The computer-readable medium of  claim 1 , further comprising extracting facial features from the still images within the dictionary. 
     
     
       11. The computer-readable medium of  claim 10 , wherein facial feature extraction comprises:
 aligning the still images based on eye locations; 
 removing a first order brightness gradient from the still images; and 
 performing histogram equalization. 
 
     
     
       12. The computer-readable medium of  claim 1 , further comprising:
 calculating an appearance affinity, a coefficient affinity, and a co-occurrence affinity for the first face track and a second face track; 
 converting the appearance affinity, the coefficient affinity, and the co-occurrence affinity into probability values using a standard sigmoid function; and 
 combining the probability values using a weighted mean equation to obtain a similarity matrix; 
 normalizing the similarity matrix to obtain a transition probability matrix; 
 propagating the transition probability matrix and confidence values to subsequent face tracks to obtain a final face identification. 
 
     
     
       13. A non-transitory computer-readable medium having stored thereon computer-executable instructions that, in response to being executed by a computing device, cause the computing device to perform operations comprising:
 receiving a video clip having a plurality of frames; 
 receiving a dictionary of still images of a plurality of faces having known identities; 
 detecting a first face within the video clip; 
 tracking the first face over the plurality of frames of the video clip to obtain a first face track; 
 computing a mean of the first face track; 
 performing an l 1 -minimization on the mean of the first face track under a predefined sparsity constraint; 
 computing class probabilities to establish an initial identity of the first face and a confidence value, wherein the initial identity of the first face is selected from the dictionary; 
 calculating affinity metrics for the first face track and a second face track; 
 fusing the affinity metrics to obtain a similarity matrix; 
 normalizing the similarity matrix to obtain a transition probability matrix; and 
 propagating the transition probability matrix and confidence values to subsequent face tracks to obtain a final face identification and a final confidence value for each face track. 
 
     
     
       14. The computer-readable medium of  claim 13 , wherein the affinity metrics are selected from the group consisting of an appearance affinity, a coefficient affinity, and a co-occurrence affinity. 
     
     
       15. The computer-readable medium of  claim 13 , wherein a single coefficient vector is imposed on the plurality of frames within the first face track. 
     
     
       16. The computer-readable medium of  claim 13 , further comprising ending the first face track in response to the first face not being detected in a predetermined number of frames following a frame in which the first face was initially detected. 
     
     
       17. The computer-readable medium of  claim 13 , wherein facial features are extracted from the still images within the dictionary by aligning the still images based on eye locations, removing a first order brightness gradient from the still images, and performing histogram equalization. 
     
     
       18. The computer-readable medium of  claim 13 , wherein local histograms are used to determine whether a second face is associated with the first face track. 
     
     
       19. The computer-readable medium of  claim 13 , further comprising:
 detecting a second face within the video clip; 
 comparing location and appearance metrics of the second face and the first face; and 
 responsive to the location and appearance metrics of the second face and the first face exhibiting a predefined level of similarity, associating the second face with the first face track. 
 
     
     
       20. The computer-readable medium of  claim 13 , further comprising obtaining a global histogram for a frame and using the global histogram to evaluate whether the frame is associated with the face track.