Emotion-aw are reactive interface

Abstract

A computer-implemented method of providing an emotion-aware reactive interface in an electronic device includes receiving an image of a user as an input and identifying a multi-modal non-verbal cue in the image. The method further includes interpreting the multi-modal non-verbal cue to determine a categorization and outputting a reactive interface event determined based on the categorization.

What is claimed is: 
     
       1. A computer-implemented method of providing an emotion-aware reactive interface in an electronic device, the method comprising:
 receiving, by a processor, image data from an image captured by a camera of the electronic device as an input via a user interface of an application on a display of the electronic device; 
 analyzing the image data, wherein the image comprises facial features and a plurality of subjects in the image to identify one or more subjects associated with the facial features, wherein a plurality of multi-modal non-verbal cues are associated with the one or more subjects corresponding to the facial features; 
 prioritizing multi-modal non-verbal cues of the plurality of multi-modal non-verbal cues based on a size and a position of the one or more subjects associated with the multi-modal non-verbal cues within the captured image and determining a first multi-modal non-verbal cue to have a higher priority relative to other multi-modal non-verbal cues associated with the captured image; 
 determining a categorization of the first multi-modal non-verbal cue, wherein the categorization is determined from a set of categorizations defined in an interpretation layer; and 
 outputting a reactive interface event determined based on the first multi-modal non-verbal cue and its categorization, 
 wherein the reactive interface event is consumed by the application to provide one or more selectable items of graphic content on the user interface of the application provided on the display of the electronic device. 
 
     
     
       2. The method of  claim 1 , wherein the categorization of the first multi-modal non-verbal cue is associated with at least one of an emotion, a gesture, a text, a scene, or a context. 
     
     
       3. The method of  claim 1 , further comprising:
 determining a second multi-modal non-verbal cue to have a higher priority relative to other multi-modal non-verbal cues associated with the captured image; 
 determining a second categorization of the second multi-modal non-verbal cue; and 
 outputting the reactive interface event determined based in part on the second multi-modal non-verbal cue and the second categorization. 
 
     
     
       4. The method of  claim 1 , wherein the plurality of multi-modal non-verbal cues comprise at least one of a facial expression, a pose, a gesture, a text, an object, or a scene. 
     
     
       5. The method of  claim 1 , wherein the reactive interface event is utilized by an application on the electronic device to provide at least one of a background effect, a pop-up image, an avatar, or a visualization in a visual interface of the application. 
     
     
       6. An apparatus comprising:
 a display; 
 a camera; and 
 a processor configured to:
 receive, by the processor, via a user interface of an application on the display, image data from an image captured by the camera of the apparatus as an input, 
 analyze the image data comprising facial features with a plurality of subjects in the image to identify one or more subjects associated with the facial features, wherein a plurality of multi-modal non-verbal cues are associated with the one or more subjects corresponding to the facial features, 
 prioritize multi-modal non-verbal cues of the plurality of multi-modal non-verbal cues based on a size and a position of the plurality of subjects associated with the multi-modal non-verbal cues within the captured image and determine a first multi-modal non-verbal cue to have a higher priority relative to other multi-modal non-verbal cues associated with the captured image, 
 determine a categorization of the first multi-modal non-verbal cue, wherein the categorization is determined from a set of categorizations defined in an interpretation layer, and 
 output a reactive interface event determined based on the first multi-modal non-verbal cue and its categorization, 
 
 wherein the reactive interface event is consumed by the application to provide one or more selectable items of graphic content on the user interface of the application provided on the display. 
 
     
     
       7. The apparatus of  claim 6 , wherein the categorization of the first multi-modal non-verbal cue is associated with at least one of an emotion, a gesture, a text, a scene, or a context. 
     
     
       8. The apparatus of  claim 6 , wherein the processor is further configured to:
 determine a second multi-modal non-verbal cue to have a higher priority relative to other multi-modal non-verbal cues associated with the captured image, 
 determine a second categorization of the second multi-modal non-verbal cue, and 
 output the reactive interface event determined based in part on the second multi-modal non-verbal cue and the second categorization. 
 
     
     
       9. The apparatus of  claim 6 , wherein the plurality of multi-modal non-verbal cues comprise at least one of a facial expression, a pose, a gesture, a text, an object, or a scene. 
     
     
       10. The apparatus of  claim 6 , wherein the reactive interface event is utilized by an application on the apparatus to provide at least one of a background effect, a pop-up image, an avatar, or a visualization in a visual interface of the application. 
     
     
       11. A non-transitory computer-readable medium comprising program code, which when executed by a processor, causes the processor to:
 receive, by the processor, via a user interface of an application on a display of an electronic device, image data from an image captured by a camera of the electronic device as an input, 
 analyze the image data comprising facial features with a plurality of subjects in the image to identify one or more subjects associated with the facial features, wherein a plurality of multi-modal non-verbal cues are associated with the one or more subjects corresponding to the facial features, 
 prioritize multi-modal non-verbal cues of the plurality of multi-modal non-verbal cues based on a size and a position of the one or more subjects associated with the multi-modal non-verbal cues within the captured image and determine a first multi-modal non-verbal cue to have a higher priority relative to other multi-modal non-verbal cues associated with the captured image, 
 determine a categorization of the first multi-modal non-verbal cue, wherein the categorization is determined from a set of categorizations defined in an interpretation layer, and 
 output a reactive interface event determined based on the first multi-modal non-verbal cue and its categorization, 
 wherein the reactive interface event is consumed by the application to provide one or more selectable items of graphic content on the user interface of the application provided on the display. 
 
     
     
       12. The non-transitory computer-readable medium of  claim 11 , wherein the categorization of the first multi-modal non-verbal cue is associated with at least one of an emotion, a gesture, a text, a scene, or a context. 
     
     
       13. The non-transitory computer-readable medium of  claim 11 , further comprising program code, which, when executed by the processor, causes the processor to:
 determine a second multi-modal non-verbal cue to have a higher priority relative to other multi-modal non-verbal cues associated with the captured image, 
 determine a second categorization of the second multi-modal non-verbal cue, and 
 output the reactive interface event determined based in part on the second multi-modal non-verbal cue and the second categorization. 
 
     
     
       14. The non-transitory computer-readable medium of  claim 11 , wherein the plurality of multi-modal non-verbal cues comprise at least one of a facial expression, a pose, a gesture, a text, an object, or a scene.