System and method for leveraging a time-series of microexpressions of users in customizing media presentation based on users# sentiments

Abstract

A system for customizing media presentation based on user's sentiments is disclosed. The system presents a media item to the user on a platform comprising a website. The system captures a first set of microexpressions of the user reacting to the media item. The system extracts a set of baseline features from the first set of microexpressions. The system determines whether the media item elicits positive or sentiment from the user. If the system determines that the media item elicits positive sentiment from the user, the system classifies the media item into a first class of media items that elicit positive sentiment from the user. The system adjusts contents of the platform to include media items from the first class of media items.

The invention claimed is: 
     
       1. A system for customizing a media presentation based on user's sentiments, comprising:
 a memory operable to store a plurality of media items comprising at least one of an image, a video, an audio, and a text; and 
 a processor, operably coupled with the memory, configured to:
 present, at a first timestamp, a first media item from the plurality of media items to the user on a particular platform, wherein the particular platform comprises a website; 
 
 a camera, operably coupled with the processor and the memory, configured to:
 in response to presenting the first media item to the user, capture one or more first images of a user reacting to the first media item; and 
 communicate the one or more first images to the processor; 
 
 the processor is further configured to:
 capture, from the one or more first images, a first set of microexpressions of the user reacting to the first media item; 
 extract a set of baseline features from the first set of microexpressions, indicating a reaction of the user to the first media item, wherein:
 the set of baseline features represents one or more of facial features of the user and a gesture performed by the user reacting to the first media item; and 
 the reaction of the user indicates an emotional response of the user to the first media item; 
 
 based at least in part upon the set of baseline features, determine whether the first media item elicits a positive sentiment or a negative sentiment from the user, wherein:
 the positive sentiment indicates positive emotions expressed by the user, and 
 the negative sentiment indicates negative emotions expressed by the user; 
 
 determine that the first media item elicits the positive sentiment from the user; 
 in response to determining that the first media item elicits the positive sentiment from the user, classify the first media item into a first class of media items that elicit the positive sentiment from the user; 
 identify at least one other media item from the first class of media items that elicit the positive sentiment from the user, wherein the at least other media item comprises a media item that presents an augmented representation of a subject associated with the first media item; and 
 add the identified at least one other media item to the particular platform. 
 
 
     
     
       2. The system of  claim 1 , wherein determining whether the first media item elicits the positive sentiment or the negative sentiment from the user, based at least in part upon the set of baseline features, comprises:
 comparing the first set of microexpressions with microexpressions that are labeled with the positive sentiment; 
 determining whether the first set of microexpressions corresponds with the microexpressions labeled with the positive sentiment; 
 in response to determining that the first set of microexpressions corresponds with the microexpressions labeled with the positive sentiment, determining that the first set of microexpressions is associated with the positive sentiment; and 
 in response to determining that the first set of microexpressions is associated with the positive sentiment, determining that the set of baseline features is associated with the positive sentiment. 
 
     
     
       3. The system of  claim 1 , wherein:
 the processor is further configured to:
 present, at a second timestamp, a second media item from the plurality of media items to the user on the particular platform; 
 
 the camera is further configured to:
 capture one or more second images of the user reacting to the second media item; and 
 communicate the one or more second images to the processor; 
 
 the processor is further configured to:
 capture, from the one or more second images, a second set of microexpressions of the user reacting to the second media item; 
 extract a set of test features from the second set of microexpressions, indicating a reaction of the user to the second media item, wherein the set of test features represents one or more of facial features of the user and a gesture performed by the user reacting to the second media item; 
 compare the second set of microexpressions with microexpressions that are labeled with the negative sentiment; 
 determine whether the second set of microexpressions corresponds with the microexpressions labeled with the negative sentiment; 
 in response to determining that the second first set of microexpressions corresponds with the microexpressions labeled with the negative sentiment, determine that the second media item elicits the negative sentiment from the user; and 
 in response to determining that the second media item elicits the negative sentiment from the user, classify the second media item into a second class of media items that elicit the negative sentiment from the user. 
 
 
     
     
       4. The system of  claim 1 , wherein the processor is further configured to:
 present, at a third timestamp, a third media item from the plurality of media items to the user on the particular platform; 
 capture a third set of microexpressions of the user reacting to the third media item; 
 extract a second set of test features from the third set of microexpressions, indicating a reaction of the user to the third media item, wherein the second set of test features represents one or more of facial features of the user and a gesture performed by the user reacting to the third media item; 
 compare the second set of test features with the set of baseline features, wherein comparing the second set of test features with the set of baseline features comprises:
 for each test feature from the second set of test features:
 determining a first numerical value representing the test feature; 
 determining a second numerical value representing a correspond baseline feature from the set of baseline feature; 
 comparing the first numerical value with the second numerical value; 
 determining whether the first numerical value is within a threshold range from the second numerical value; and 
 in response to determining that the first numerical value is within the threshold range from the second numerical value, determining that the test feature corresponds with the corresponding baseline feature; 
 
 
 determine whether more than a threshold percentage of the second set of test features correspond with more than the threshold percentage of the set of baseline features; 
 in response to determining that the more than the threshold percentage of the second set of test features correspond with more than the threshold percentage of the set of baseline features, determine that the second set of test features corresponds with the set of baseline features; and 
 in response to determining that the second set of test features corresponds with the set of baseline features, classify the third media item into the first class of media items. 
 
     
     
       5. The system of  claim 1 , further comprising a microphone, operably coupled with the processor, configured to:
 in response to presenting the first media item to the user, capture an audio sound made by the user; and 
 communicate the audio sound to the processor; 
 the processor is further configured to extract, from the audio sound, a set of auditory microexpressions from the user reacting to the first media item, wherein:
 the set of auditory microexpressions represents auditory clues expressed by the user indicating the reaction of the user to the first media item; and 
 the set of auditory microexpressions is a portion of the first set of microexpressions. 
 
 
     
     
       6. The system of  claim 1 , wherein capturing the first set of microexpressions of the user reacting to the first media item, comprises at least one of:
 capturing facial microexpressions from the user; 
 capturing verbal auditory clues from the user comprising at least a word uttered by the user; and 
 capturing non-verbal auditory clues from the user comprising an audio sound of laughing. 
 
     
     
       7. The system of  claim 1 , wherein the set of baseline features further represents one or more of a movement of the user and an audio sound made by the user. 
     
     
       8. A method for customizing a media presentation based on user's sentiments, comprising:
 presenting, at a first timestamp, a first media item from a plurality of media items to the user on a particular platform, wherein:
 the particular platform comprises a website; and 
 the plurality of media items comprises at least one of an image, a video, an audio, and a text; 
 
 in response to presenting the first media item to the user, capturing one or more first images of a user reacting to the first media item; 
 capturing, from the one or more first images, a first set of microexpressions of the user reacting to the first media item; 
 extracting a set of baseline features from the first set of microexpressions, indicating a reaction of the user to the first media item, wherein:
 the set of baseline features represents one or more of facial features of the user and a gesture performed by the user reacting to the first media item; and 
 the reaction of the user indicates an emotional response of the user to the first media item; 
 
 based at least in part upon the set of baseline features, determining whether the first media item elicits a positive sentiment or a negative sentiment from the user, wherein:
 the positive sentiment indicates positive emotions expressed by the user, and 
 the negative sentiment indicates negative emotions expressed by the user; 
 
 determining that the first media item elicits the positive sentiment from the user; 
 in response to determining that the first media item elicits the positive sentiment from the user, classifying the first media item into a first class of media items that elicit the positive sentiment from the user; 
 identifying at least one other media item from the first class of media items that elicit the positive sentiment from the user, wherein the at least other media item comprises a media item that presents an augmented representation of a subject associated with the first media item; and 
 adding the identified at least one other media item to the particular platform. 
 
     
     
       9. The method of  claim 8 , wherein determining whether the first media item elicits the positive sentiment or the negative sentiment from the user, based at least in part upon the set of baseline features, comprises:
 comparing the first set of microexpressions with microexpressions that are labeled with the positive sentiment; 
 determining whether the first set of microexpressions corresponds with the microexpressions labeled with the positive sentiment; 
 in response to determining that the first set of microexpressions corresponds with the microexpressions labeled with the positive sentiment, determining that the first set of microexpressions is associated with the positive sentiment; and 
 in response to determining that the first set of microexpressions is associated with the positive sentiment, determining that the set of baseline features is associated with the positive sentiment. 
 
     
     
       10. The method of  claim 8 , further comprising:
 presenting, at a second timestamp, a second media item from the plurality of media items to the user on the particular platform; 
 capturing one or more second images of the user reacting to the second media item; 
 capturing, from the one or more second images, a second set of microexpressions of the user reacting to the second media item; 
 extracting a set of test features from the second set of microexpressions, indicating a reaction of the user to the second media item, wherein the set of test features represents one or more of facial features of the user and a gesture performed by the user reacting to the second media item; 
 comparing the second set of microexpressions with microexpressions that are labeled with the negative sentiment; 
 determining whether the second set of microexpressions corresponds with the microexpressions labeled with the negative sentiment; 
 in response to determining that the second first set of microexpressions corresponds with the microexpressions labeled with the negative sentiment, determining that the second media item elicits the negative sentiment from the user; and 
 in response to determining that the second media item elicits the negative sentiment from the user, classifying the second media item into a second class of media items that elicit the negative sentiment from the user. 
 
     
     
       11. The method of  claim 8 , further comprising:
 presenting, at a third timestamp, a third media item from the plurality of media items to the user on the particular platform; 
 capturing a third set of microexpressions of the user reacting to the third media item; 
 extracting a second set of test features from the third set of microexpressions, indicating a reaction of the user to the third media item, wherein the second set of test features represents one or more of facial features of the user and a gesture performed by the user reacting to the third media item; 
 comparing the second set of test features with the set of baseline features, wherein comparing the second set of test features with the set of baseline features comprises:
 for each test feature from the second set of test features:
 determining a first numerical value representing the test feature; 
 determining a second numerical value representing a correspond baseline feature from the set of baseline feature; 
 comparing the first numerical value with the second numerical value; 
 determining whether the first numerical value is within a threshold range from the second numerical value; and 
 in response to determining that the first numerical value is within the threshold range from the second numerical value, determining that the test feature corresponds with the corresponding baseline feature; 
 
 
 determining whether more than a threshold percentage of the second set of test features correspond with more than the threshold percentage of the set of baseline features; 
 in response to determining that the more than the threshold percentage of the second set of test features correspond with more than the threshold percentage of the set of baseline features, determining that the second set of test features corresponds with the set of baseline features; and 
 in response to determining that the second set of test features corresponds with the set of baseline features, classifying the third media item into the first class of media items. 
 
     
     
       12. The method of  claim 8 , further comprising:
 in response to presenting the first media item to the user, capturing an audio sound made by the user; and 
 extracting, from the audio sound, a set of auditory microexpressions from the user reacting to the first media item, wherein:
 the set of auditory microexpressions represents auditory clues expressed by the user indicating the reaction of the user to the first media item; and 
 the set of auditory microexpressions is a portion of the first set of microexpressions. 
 
 
     
     
       13. The method of  claim 8 , wherein capturing the first set of microexpressions of the user reacting to the first media item, comprises at least one of:
 capturing facial microexpressions from the user; 
 capturing verbal auditory clues from the user comprising at least a word uttered by the user; and 
 capturing non-verbal auditory clues from the user comprising an audio sound of laughing. 
 
     
     
       14. The method of  claim 8 , further comprising:
 determining a first sentiment expressed by a first set of users reacting to the first media item; 
 determining a second sentiment expressed by a second set of users reacting to the first media item; 
 determining that the first sentiment is an anomaly compared to the second sentiment; and 
 triggering a detectable action to address the anomaly, comprising communicating a message to authorities supervising the first set of users indicating the anomaly associated with the first set of users reacting to the first media item. 
 
     
     
       15. A computer program comprising executable instructions stored in a non-transitory computer-readable medium that when executed by a processor causes the processor to:
 present, at a first timestamp, a first media item from the plurality of media items to the user on a particular platform, wherein:
 the particular platform comprises a website; and 
 the plurality of media items comprises at least one of an image, a video, an audio, and a text; 
 
 in response to presenting the first media item to the user, capture one or more first images of a user reacting to the first media item; 
 capture, from the one or more first images, a first set of microexpressions of the user reacting to the first media item; 
 extract a set of baseline features from the first set of microexpressions, indicating a reaction of the user to the first media item, wherein:
 the set of baseline features represents one or more of facial features of the user and a gesture performed by the user reacting to the first media item; and 
 the reaction of the user indicates an emotional response of the user to the first media item; 
 
 based at least in part upon the set of baseline features, determine whether the first media item elicits a positive sentiment or a negative sentiment from the user, wherein:
 the positive sentiment indicates positive emotions expressed by the user, and 
 the negative sentiment indicates negative emotions expressed by the user; 
 
 determine that the first media item elicits the positive sentiment from the user; 
 in response to determining that the first media item elicits the positive sentiment from the user, classify the first media item into a first class of media items that elicit the positive sentiment from the user; 
 identify at least one other media item from the first class of media items that elicit the positive sentiment from the user, wherein the at least other media item comprises a media item that presents an augmented representation of a subject associated with the first media item; and 
 add the identified at least one other media item to the particular platform. 
 
     
     
       16. The computer program of  claim 15 , wherein determining whether the first media item elicits the positive sentiment or the negative sentiment from the user, based at least in part upon the set of baseline features, comprises:
 comparing the first set of microexpressions with microexpressions that are labeled with the positive sentiment; 
 determining whether the first set of microexpressions corresponds with the microexpressions labeled with the positive sentiment; 
 in response to determining that the first set of microexpressions corresponds with the microexpressions labeled with the positive sentiment, determining that the first set of microexpressions is associated with the positive sentiment; and 
 in response to determining that the first set of microexpressions is associated with the positive sentiment, determining that the set of baseline features is associated with the positive sentiment. 
 
     
     
       17. The computer program of  claim 15 , wherein the instructions when executed by the processor, further cause the processor to:
 present, at a second timestamp, a second media item from the plurality of media items to the user on the particular platform; 
 capture one or more second images of the user reacting to the second media item; 
 capture, from the one or more second images, a second set of microexpressions of the user reacting to the second media item; 
 extract a set of test features from the second set of microexpressions, indicating a reaction of the user to the second media item, wherein the set of test features represents one or more of facial features of the user and a gesture performed by the user reacting to the second media item; 
 compare the second set of microexpressions with microexpressions that are labeled with the negative sentiment; 
 determine whether the second set of microexpressions corresponds with the microexpressions labeled with the negative sentiment; 
 in response to determining that the second first set of microexpressions corresponds with the microexpressions labeled with the negative sentiment, determine that the second media item elicits the negative sentiment from the user; and 
 in response to determining that the second media item elicits the negative sentiment from the user, classify the second media item into a second class of media items that elicit the negative sentiment from the user. 
 
     
     
       18. The computer program of  claim 15 , wherein the instructions when executed by the processor, further cause the processor to:
 present, at a third timestamp, a third media item from the plurality of media items to the user on the particular platform; 
 capture a third set of microexpressions of the user reacting to the third media item; 
 extract a second set of test features from the third set of microexpressions, indicating a reaction of the user to the third media item, wherein the second set of test features represents one or more of facial features of the user and a gesture performed by the user reacting to the third media item; 
 compare the second set of test features with the set of baseline features, wherein comparing the second set of test features with the set of baseline features comprises:
 for each test feature from the second set of test features:
 determining a first numerical value representing the test feature; 
 determining a second numerical value representing a correspond baseline feature from the set of baseline feature; 
 comparing the first numerical value with the second numerical value; 
 determining whether the first numerical value is within a threshold range from the second numerical value; and 
 in response to determining that the first numerical value is within the threshold range from the second numerical value, determining that the test feature corresponds with the corresponding baseline feature; 
 
 
 determine whether more than a threshold percentage of the second set of test features correspond with more than the threshold percentage of the set of baseline features;
 in response to determining that the more than the threshold percentage of the second set of test features correspond with more than the threshold percentage of the set of baseline features, determine that the second set of test features corresponds with the set of baseline features; and 
 
 is response to determining that the second set of test features corresponds with the set of baseline features, classify the third media item into the first class of media items. 
 
     
     
       19. The computer program of  claim 15 , further comprising a microphone, operably coupled with the processor, configured to:
 in response to presenting the first media item to the user, capture an audio sound made by the user; and 
 communicate the audio sound to the processor; 
 the processor is further configured to extract, from the audio sound, a set of auditory microexpressions from the user reacting to the first media item, wherein:
 the set of auditory microexpressions represents auditory clues expressed by the user indicating the reaction of the user to the first media item; and 
 the set of auditory microexpressions is a portion of the first set of microexpressions. 
 
 
     
     
       20. The computer program of  claim 15 , wherein capturing the first set of microexpressions of the user reacting to the first media item, comprises at least one of:
 capturing facial microexpressions from the user; 
 capturing verbal auditory clues from the user comprising at least a word uttered by the user; and 
 capturing non-verbal auditory clues from the user comprising an audio sound of laughing.