Systems and methods for determining emotions based on user gestures

Abstract

Various embodiments of the invention allow to detect and analyze gestures, such as tapping and swiping patterns, that a user performs in the process of interacting with a computing device to determine the user's mood therefrom, so as to initiate an appropriate response. In certain embodiments, this is accomplished, without requiring labeled training data, by monitoring a user-device interaction via sensors and analyzing the sensor data based on contextual data via a processor to determine a gesture and one or more properties associated with an emotional state of the user. A response is generated based on the identified emotional state.

What is claimed is: 
     
       1. A method for determining user emotional states, the method comprising:
 receiving, via one or more standard inputs of a device, an initial set of input data comprising a first set of contextual data and gesture data relating to a user-device interaction, the initial set of input data comprising generic data and being unlabeled; 
 generating derived data from the generic data; 
 based on at least one of the generic data or the derived data, identifying a first set of patterns; 
 assigning to the first set of patterns a first set of characteristics indicative of a generic user emotional state; 
 applying the first set of characteristics to a generic model to generate a generic emotional state profile; 
 collecting user-specific input data comprising a second set of contextual data and gesture data regarding interactions between a user and a computing device; 
 clustering the user-specific input data based on, at least, a second set of characteristics; 
 applying the second set of characteristics to a user-specific model to generate a user-specific emotional state profile regarding the interactions, wherein generating the user-specific emotional state profile comprises comparing the user's behavior against the generic model; 
 feeding back the clustered user-specific input data as inputs to the user-specific model to refine the user-specific emotional state profile by adjusting a threshold value in the user-specific model by assigning to a parameter of the user-specific emotional state profile a weighing factor that is based on a relationship between the gesture data and the second set of contextual data; and 
 using the user-specific emotional state profile and the second set of contextual data to identify a same user emotional state in at least two different environments. 
 
     
     
       2. The method according to  claim 1 , further comprising correlating at least some of the gesture data and at least some of the contextual data with a user emotional state. 
     
     
       3. The method according to  claim 1  further comprising inputting collected input data comprising contextual data and gesture data into the user emotional state profile to determine a user emotional state during a time period in which at least some of the collected input data was obtained. 
     
     
       4. The method according to  claim 1 , further comprising updating the user-specific emotional state profile based on a second user-specific profile that is assigned to another user. 
     
     
       5. The method according to  claim 1 , wherein the user-specific emotional state profile comprises a trained set of characteristics. 
     
     
       6. The method according to  claim 1 , wherein the gesture data comprises information about at least one of a swipe pattern, a pressure exerted by a finger, a finger size, a speed of the gesture, a start or stop location, and a change in intensity of the gesture, and wherein contextual data comprises information about at least one of an activity of a user, a location of a user, and other nearby users. 
     
     
       7. The method according to  claim 1 , further comprising triggering a reference emotional state in a training session. 
     
     
       8. The method according to  claim 7 , wherein triggering the reference emotional state comprises one of preventing the computing device from responding to a user input and causing a delay in responding to the user input. 
     
     
       9. The method according to  claim 1 , further comprising applying a testing procedure to evaluate one of a cognitive ability and a performance prior to permitting a user-device interaction. 
     
     
       10. A method for determining a user emotional state, the method comprising:
 receiving, via one or more standard inputs of a device, an initial set of input data comprising a first set of contextual data and gesture data relating to a user-device interaction, the initial set of input data comprising generic data and being unlabeled; 
 based on at least the generic data, identifying a first set of patterns associated with a first set of characteristics that are indicative of a generic user emotional state; 
 applying the first set of characteristics to a generic model to generate a generic emotional state profile; 
 collecting user-specific input data comprising a second set of contextual data and gesture data regarding interactions between a user and a computing device; 
 clustering the user-specific input data based on, at least, a second set of characteristics; 
 applying the second set of characteristics to a user-specific model to generate a user-specific emotional state profile, wherein generating the user-specific emotional state profile comprises comparing the user's behavior against the generic model; 
 feeding back the clustered user-specific input data as inputs to the user-specific model to refine the user-specific emotional state profile by adjusting a threshold value in the user-specific model; 
 using the user-specific emotional state profile and the second set of contextual data to identify a user emotional state; 
 outputting the user emotional state; and 
 generating a response based on the user emotional state. 
 
     
     
       11. The method according to  claim 10 , wherein the response is designed to do at least one of affecting a change in the user's mood, reengaging the user with a task, and adjusting a difficulty level of the task. 
     
     
       12. The method according to  claim 10 , further comprising monitoring the user-device interaction by using sensors communicatively coupled to an analysis module. 
     
     
       13. The method according to  claim 10 , wherein the user-specific emotional state profile is generated in a training session by clustering the input data based on characteristics associated with emotional states. 
     
     
       14. The method according to  claim 10 , further comprising refining the user-specific emotional state profile based on feedback data. 
     
     
       15. A system for determining an emotional state of a user of a computing system, the system comprising:
 one or more standard inputs of a device to collect an initial set of input data related to gesture data and a first set of contextual data, the gesture data relates to a user-device interaction between a user and a computing device, the initial set of input data comprising generic data and requiring no labeling; 
 an analysis module that performs steps comprising:
 based on at least the generic data, identifying a first set of patterns associated with a first set of characteristics that are indicative of a generic user emotional state; 
 applying the first set of characteristics to a generic model to generate a generic emotional state profile; 
 collecting user-specific input data comprising a second set of contextual data and gesture data regarding interactions between a user and a computing device; 
 clustering the user-specific input data based on, at least, a second set of characteristics; 
 applying the second set of characteristics to a user-specific model to generate a user-specific emotional state profile, wherein generating the user-specific emotional state profile comprises comparing the user's behavior against the generic model; 
 feeding back the clustered user-specific input data as inputs to the user-specific model to refine the user-specific emotional state profile by adjusting a threshold value in the user-specific model; and 
 using the user-specific emotional state profile and the second set of contextual data to identify user emotional state; and 
 
 a response generator coupled to the analysis module to receive the user emotional state and generate a response based on the user emotional state. 
 
     
     
       16. The system according to  claim 15 , wherein the analysis module further comprises a processing module coupled to the one or more standard inputs of the device, the processing module associates the one or more gestures and the contextual data with the emotional state and generates an output signal representative of the user emotional state. 
     
     
       17. The system according to  claim 16 , wherein the processing module is configured to compare the one or more gestures to one or more reference gestures. 
     
     
       18. The system according to  claim 16 , wherein the processing module is configured to use a machine learning process. 
     
     
       19. The system according to  claim 15 , wherein the one or more standard inputs of the device are implemented in at least one of a pointing device, a keyboard, and a touchscreen and comprise an accelerometer that detects one of an orientation and a speed of a device housing at least part of the computing system. 
     
     
       20. The system according to  claim 15 , wherein the analysis module is configured to cluster, in a training phase, gesture data based on a characteristic pattern that is associated with an emotional state parameter.