Identifying Facial Expressions in Acquired Digital Images

Abstract

A face is detected and identified within an acquired digital image. One or more features of the face is/are extracted from the digital image, including two independent eyes or subsets of features of each of the two eyes, or lips or partial lips or one or more other mouth features and one or both eyes, or both. A model including multiple shape parameters is applied to the two independent eyes or subsets of features of each of the two eyes, and/or to the lips or partial lips or one or more other mouth features and one or both eyes. One or more similarities between the one or more features of the face and a library of reference feature sets is/are determined. A probable facial expression is identified based on the determining of the one or more similarities.

What is claimed is: 
     
         1 . A method of initiating one or more further actions based on recognizing a facial expression and an identity of a face within a digital image, comprising:
 using a processor;   acquiring a digital image;   detecting a face within the digital image;   applying a facial model to the face to identify the face or a facial expression of the face, or both, within the digital image;   separately extracting one or more features of the face within the digital image, including two independent eyes or subsets of features of each of the two eyes, or lips or partial lips or one or more other mouth features and one or both eyes, or both;   applying a feature model including multiple shape parameters to said two independent eyes or subsets of features of each of the two eyes, or to said lips or partial lips or one or more other mouth features and one or both eyes, or both;   determining one or more similarities between the one or more features of the face and a library of reference feature sets;   identifying a probable facial expression based on the determining of the one or more similarities; and   initiating one or more further actions based on the identity of the face and the probable facial expression.   
     
     
         2 . The method of  claim 1 , wherein the one or more features comprise one or more geometric features. 
     
     
         3 . The method of  claim 2 , wherein the one or more geometric features comprise one or more shapes, deformations or locations of facial components, or pose variations, or combinations thereof. 
     
     
         4 . The method of  claim 1 , wherein the extracting comprises extracting one or more appearance features. 
     
     
         5 . The method of  claim 4 , wherein the one or more appearance features comprise one or more skin texture changes. 
     
     
         6 . The method of  claim 5 , wherein the one or more skin texture changes comprise one or more furrows, bulges, expression wrinkles, or illumination variations, or blushing, or combinations thereof. 
     
     
         7 . The method of  claim 1 , wherein the library comprises face features of faces captured in multiple pose variations or under multiple directional illumination conditions, or both. 
     
     
         8 . The method of  claim 1 , further comprising categorizing the facial expression as indicating surprise, fear, happiness, anger, neutral, sadness, disgust, or a combination thereof. 
     
     
         9 . The method of  claim 1 , further comprising adapting a game difficulty based on the identified probable facial expression. 
     
     
         10 . The method of  claim 1 , further comprising adapting a game workflow based on the identified probable facial expression. 
     
     
         11 . The method of  claim 1 , further comprising mirroring the identified probable facial expression in an avatar within a game display. 
     
     
         12 . The method of  claim 1 , wherein the extracting separately of the one or more features of the face is performed before the identifying of the face. 
     
     
         13 . The method of  claim 1 , wherein the model comprises an Active Appearance Model (AAM). 
     
     
         14 . The method of  claim 1 , wherein the multiple shape parameters comprise seven or less shape parameters. 
     
     
         15 . The method of  claim 1 , wherein the multiple shape parameters comprise six or seven shape parameters. 
     
     
         16 . The method of  claim 1 , wherein the initiating one or more further actions comprises adjusting the feature model to change the probable facial expression based on the identity of the face. 
     
     
         17 . The method of  claim 16 , wherein the initiating one or more further actions further comprises adjusting the feature model to change the probable facial expression based on the probable facial expression. 
     
     
         18 . The method of  claim 1 , wherein the initiating one or more further actions further comprises adjusting the feature model to change the probable facial expression based on the probable facial expression. 
     
     
         19 . A digital image acquisition device configured to recognize a facial expression or an identity of a face, or both, comprising:
 a lens and image sensor for capturing a digital image;   a processor;   a face detection module configured to program the processor to detect a face within the digital image;   a facial recognition module configured to program the processor to apply a facial model to identify the face within the digital image;   a feature extraction module configured to program the processor to extract one or more features of the face within the digital image, including two independent eyes or subsets of features of each of the two eyes, or lips or partial lips or one or more other mouth features and one or both eyes, or both;   a feature model application module configured to program the processor to apply a feature model including multiple shape parameters to said two independent eyes or subsets of features of each of the two eyes, or to said lips or partial lips or one or more other mouth features and one or both eyes, or both;   a classification module configured to program the processor to determine one or more similarities between the one or more features of the face and a library of reference feature sets; and   a facial expression identification module configured to program the processor to identify a probable facial expression based on the determining of the one or more similarities, and   wherein the device is further configured to program the processor to initiate one or more further actions based on the identity of the face and the probable facial expression.   
     
     
         20 . The device of  claim 19 , wherein the one or more features comprise one or more geometric features. 
     
     
         21 . The device of  claim 20 , wherein the one or more geometric features comprise one or more shapes, deformations or locations of facial components, or pose variations, or combinations thereof. 
     
     
         22 . The device of  claim 19 , wherein the extracting comprises extracting one or more appearance features. 
     
     
         23 . The device of  claim 22 , wherein the one or more appearance features comprise one or more skin texture changes. 
     
     
         24 . The device of  claim 23 , wherein the one or more skin texture changes comprise one or more furrows, bulges, expression wrinkles, or illumination variations, or blushing, or combinations thereof. 
     
     
         25 . The device of  claim 19 , wherein the library comprises face features of faces captured in multiple pose variations or under multiple directional illumination conditions, or both. 
     
     
         26 . The device of  claim 19 , further comprising a categorization module configured to program the processor to classify the facial expression as indicating surprise, fear, happiness, anger, neutral, sadness, disgust, or a combination thereof. 
     
     
         27 . The device of  claim 19 , further comprising a game adaptation module configured to adjust a game difficulty based on the identified probable facial expression. 
     
     
         28 . The device of  claim 19 , further comprising a game adaptation module configured to program the processor to adjust a game workflow based on the identified probable facial expression. 
     
     
         29 . The device of  claim 19 , further comprising an avatar expression module configured to program the processor to adjust an expression of an avatar in a game display to mirror the identified probable facial expression. 
     
     
         30 . The device of  claim 19 , wherein the extracting separately of the one or more features of the face is performed before the identifying of the face. 
     
     
         31 . The device of  claim 19 , wherein the model comprises an Active Appearance Model (AAM). 
     
     
         32 . The device of  claim 19 , wherein the multiple shape parameters comprise seven or less shape parameters. 
     
     
         33 . The device of  claim 19 , wherein the multiple shape parameters comprise six or seven shape parameters. 
     
     
         34 . The device of  claim 19 , wherein the device is programmed to adjust the feature model to change the probable facial expression based on the identity of the face. 
     
     
         35 . The device of  claim 34 , wherein the device is programmed to adjust the feature model to change the probable facial expression based on the probable facial expression. 
     
     
         36 . The device of  claim 19 , wherein the device is programmed to adjust the feature model to change the probable facial expression based on the probable facial expression. 
     
     
         37 . One or more non-transitory computer-readable media having code embedded therein for programming a processor to perform a method of initiating one or more further actions based on recognizing a facial expression and an identity of a face within an acquired digital image, wherein the method comprises:
 detecting a face within the acquired digital image;   applying a facial model to the face to identify the face or a facial expression of the face, or both, within the digital image;   separately extracting one or more features of the face within the digital image, including two independent eyes or subsets of features of each of the two eyes, or lips or partial lips or one or more other mouth features and one or both eyes, or both;   applying a feature model including multiple shape parameters to said two independent eyes or subsets of features of each of the two eyes, or to said lips or partial lips or one or more other mouth features and one or both eyes, or both;   determining one or more similarities between the one or more features of the face and a library of reference feature sets;   identifying a probable facial expression based on the determining of the one or more similarities; and   initiating one or more further actions based on the identity of the face and the probable facial expression.   
     
     
         38 . The one or more computer-readable media of  claim 37 , wherein the one or more features comprise one or more geometric features. 
     
     
         39 . The one or more computer-readable media of  claim 38 , wherein the one or more geometric features comprise one or more shapes, deformations or locations of facial components, or pose variations, or combinations thereof. 
     
     
         40 . The one or more computer-readable media of  claim 37 , wherein the extracting comprises extracting one or more appearance features. 
     
     
         41 . The one or more computer-readable media of  claim 40 , wherein the one or more appearance features comprise one or more skin texture changes. 
     
     
         42 . The one or more computer-readable media of  claim 41 , wherein the one or more skin texture changes comprise one or more furrows, bulges, expression wrinkles, or illumination variations, or blushing, or combinations thereof. 
     
     
         43 . The one or more computer-readable media of  claim 37 , wherein the library comprises face features of faces captured in multiple pose variations or under multiple directional illumination conditions, or both. 
     
     
         44 . The one or more computer-readable media of  claim 37 , wherein the method further comprises categorizing the facial expression as indicating surprise, fear, happiness, anger, neutral, sadness, disgust, or a combination thereof. 
     
     
         45 . The one or more computer-readable media of  claim 37 , wherein the method further comprises adapting a game difficulty based on the identified probable facial expression. 
     
     
         46 . The one or more computer-readable media of  claim 37 , wherein the method further comprises adapting a game workflow based on the identified probable facial expression. 
     
     
         47 . The one or more computer-readable media of  claim 37 , wherein the method further comprises mirroring the identified probable facial expression in an avatar within a game display. 
     
     
         48 . The one or more computer-readable media of  claim 37 , wherein the extracting separately of the one or more features of the face is performed before the identifying of the face. 
     
     
         49 . The one or more computer-readable media of  claim 37 , wherein the model comprises an Active Appearance Model (AAM). 
     
     
         50 . The one or more computer-readable media of  claim 37 , wherein the multiple shape parameters comprise seven or less shape parameters. 
     
     
         51 . The one or more computer-readable media of  claim 37 , wherein the multiple shape parameters comprise six or seven shape parameters. 
     
     
         52 . The one or more computer-readable media of  claim 37 , wherein the initiating one or more further actions comprises adjusting the feature model to change the probable facial expression based on the identity of the face. 
     
     
         53 . The one or more computer-readable media of  claim 52 , wherein the initiating one or more further actions further comprises adjusting the feature model to change the probable facial expression based on the probable facial expression. 
     
     
         54 . The one or more computer-readable media of  claim 37 , wherein the initiating one or more further actions further comprises adjusting the feature model to change the probable facial expression based on the probable facial expression.