Method and system for sentiment analysis

Abstract

Methods, systems, and computer program product for automatically performing sentiment analysis on texts, such as telephone call transcripts and electronic written communications. Disclosed techniques include, inter alia, lexicon training, handling of negations and shifters, pruning of lexicons, confidence calculation for token orientation, supervised customization, lexicon mixing, and adaptive segmentation.

What is claimed is: 
     
       1. A method comprising:
 receiving a plurality of lexicons, wherein each of said plurality of lexicons comprises a plurality of tokens associated with a specified domain, and wherein each of said tokens has an associated sentiment orientation and sentiment score; 
 automatically identifying, in said plurality of lexicons:
 (i) mutual tokens which exist in more than one of the plurality of lexicons, wherein each of said mutual tokens is assigned a sentiment score equal to a linear interpolation of each of said associated sentiment scores of said mutual token, and 
 (ii) solitary tokens which exist only in one of said plurality of lexicons, wherein each of said solitary tokens is assigned said associated sentiment score from said one of said plurality of lexicons; and 
 
 automatically generating a new lexicon in said specified domain comprising at least some of (i) and (ii). 
 
     
     
       2. The method of  claim 1 , further comprising:
 training a sentiment analysis classifier on a training set comprising at least said new lexicon; and 
 applying said sentiment analysis classifier to a text corpus. 
 
     
     
       3. The method of  claim 1 , wherein each of said associated sentiment orientations is selected from the group consisting of positive, negative, and neutral, and wherein said method further comprises automatically pruning each of said tokens having a greater probability of being associated with (i) said neutral sentiment orientation, than with (ii) said positive and said negative sentiment orientations. 
     
     
       4. The method of  claim 1 , wherein said interpolation comprises weighting each of said tokens based, at least in part, on a weighted average, wherein the weights in the weighted average are assigned based on a number of occurrences of each of said tokens in said plurality of lexicons. 
     
     
       5. The method of  claim 1 , wherein said interpolation comprises weighting each of said tokens based, at least in part, on a weighted average, wherein the weights in the weighted average are assigned based on a confidence score of each of said tokens. 
     
     
       6. A system comprising:
 at least one hardware processor; and 
 a non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions executable by the at least one hardware processor to:
 receive a plurality of lexicons, wherein each of said plurality of lexicons comprises a plurality of tokens associated with a specified domain, and wherein each of said tokens has an associated sentiment orientation and sentiment score, 
 
 automatically identify, in said plurality of lexicons:
 (i) mutual tokens which exist in more than one of the plurality of lexicons, wherein each of said mutual tokens is assigned a sentiment score equal to a linear interpolation of each of said associated sentiment scores of said mutual token, and 
 (ii) solitary tokens which exist only in one of said plurality of lexicons, wherein each of said solitary tokens is assigned said associated sentiment score from said one of said plurality of lexicons, and 
 
 automatically generate a new lexicon in said specified domain comprising at least some of (i) and (ii). 
 
     
     
       7. The system of  claim 6 , wherein said program instructions are further executable to:
 train a sentiment analysis classifier on a training set comprising at least said new lexicon; and 
 apply said sentiment analysis classifier to a text corpus. 
 
     
     
       8. The system of  claim 6 , wherein each of said associated sentiment orientations is selected from the group consisting of positive, negative, and neutral, and wherein said program instructions are further executable to automatically prune each of said tokens having a greater probability of being associated with (i) said neutral sentiment orientation, than with (ii) said positive and said negative sentiment orientations. 
     
     
       9. The system of  claim 6 , wherein said interpolation comprises weighting each of said tokens based, at least in part, on a weighted average, wherein the weights in the weighted average are assigned based on a number of occurrences of each of said tokens in said plurality of lexicons. 
     
     
       10. The system of  claim 6 , wherein said interpolation comprises weighting each of said tokens based, at least in part, on a weighted average, wherein the weights in the weighted average are assigned based on a confidence score of each of said tokens. 
     
     
       11. A computer program product comprising a non-transitory computer-readable storage medium having program instructions embodied therewith, the program instructions executable by at least one hardware processor to:
 receive a plurality of lexicons, wherein each of said plurality of lexicons comprises a plurality of tokens associated with a specified domain, and wherein each of said tokens has an associated sentiment orientation and sentiment score; 
 automatically identify, in said plurality of lexicons:
 (i) mutual tokens which exist in more than one of the plurality of lexicons, wherein each of said mutual tokens is assigned a sentiment score equal to a linear interpolation of each of said associated sentiment scores of said mutual token, and 
 (ii) solitary tokens which exist only in one of said plurality of lexicons, wherein each of said solitary tokens is assigned said associated sentiment score from said one of said plurality of lexicons; and 
 
 automatically generate a new lexicon in said specified domain comprising at least some of (i) and (ii). 
 
     
     
       12. The computer program product of  claim 11 , wherein said program instructions are further executable to:
 train a sentiment analysis classifier on a training set comprising at least said new lexicon; and 
 apply said sentiment analysis classifier to a text corpus. 
 
     
     
       13. The computer program product of  claim 11 , wherein each of said associated sentiment orientations is selected from the group consisting of positive, negative, and neutral, and wherein said program instructions are further executable to automatically prune each of said tokens having a greater probability of being associated with (i) said neutral sentiment orientation, than with (ii) said positive and said negative sentiment orientations. 
     
     
       14. The computer program product of  claim 11 , wherein said interpolation comprises weighting each of said tokens based, at least in part, on a weighted average, wherein the weights in the weighted average are assigned based on a number of occurrences of each of said tokens in said plurality of lexicons. 
     
     
       15. The computer program product of  claim 11 , wherein said interpolation comprises weighting each of said tokens based, at least in part, on a weighted average, wherein the weights in the weighted average are assigned based on a confidence score of each of said tokens. 
     
     
       16. A method comprising:
 receiving one or more domain-specific lexicons in a specified domain, wherein each of said domain-specific lexicons comprises a plurality of tokens having each an associated sentiment orientation and sentiment score; 
 receiving one or more non-domain specific lexicons, wherein each of said non-domain specific lexicons comprises a plurality of tokens having each an associated sentiment orientation and sentiment score; 
 automatically identifying, in all said domain-specific and non-domain specific lexicons:
 (i) a first subset of tokens which exist in one or more of the domain-specific lexicons, wherein each of said tokens in said first subset is assigned a sentiment score equal to a linear interpolation of said associated sentiment score of said token in each of said domain-specific lexicons, and 
 (ii) a second subset of tokens which do not exist in any of the domain-specific lexicons, wherein each token in said second subset is assigned a sentiment score equal to zero; and 
 
 automatically generating a new lexicon in said specified domain comprising at least some of said first and second subsets. 
 
     
     
       17. The method of  claim 16 , further comprising identifying a third subset of tokens which exist in one or more of the domain-specific lexicons and one or more of the non-domain specific lexicons, wherein each of said tokens in said third subset is assigned a sentiment score equal to a linear interpolation of said associated sentiment score of said token in each of said domain-specific and non-domain specific lexicons; wherein said new lexicon further comprises at least some of said third subset. 
     
     
       18. The method of  claim 16 , further comprising:
 training a sentiment analysis classifier on a training set comprising at least said new lexicon; and 
 applying said sentiment analysis classifier to a text corpus. 
 
     
     
       19. The method of  claim 16 , wherein each of said associated sentiment orientations is selected from the group consisting of positive, negative, and neutral, and wherein said method further comprises automatically pruning each of said tokens having a greater probability of being associated with (i) said neutral sentiment orientation, than with (ii) said positive and said negative sentiment orientations. 
     
     
       20. The method of  claim 16 , wherein said interpolation comprises weighting of each of said tokens based, at least in part, on a weighted average, wherein the weights in the weighted average are assigned based on a number of occurrences of each of said tokens in each of said domain-specific and non-domain specific lexicons. 
     
     
       21. The method of  claim 16 , wherein said interpolation comprises weighting each said token based, at least in part, on a weighted average, wherein the weights in the weighted average are assigned based on a confidence score of each of said tokens. 
     
     
       22. A system comprising:
 at least one hardware processor; and 
 a non-transitory computer-readable storage medium having stored thereon program instructions, the program instructions executable by the at least one hardware processor to:
 receive one or more domain-specific lexicons in a specified domain, wherein each of said domain-specific lexicons comprises a plurality of tokens having each an associated sentiment orientation and sentiment score, 
 receive one or more non-domain specific lexicons, wherein each of said non-domain specific lexicons comprises a plurality of tokens having each an associated sentiment orientation and sentiment score, 
 automatically identify, in all said domain-specific and non-domain specific lexicons:
 (i) a first subset of tokens which exist in one or more of the domain-specific lexicons, wherein each of said tokens in said first subset is assigned a sentiment score equal to a linear interpolation of said associated sentiment score of said token in each of said domain-specific lexicons, and 
 (ii) a second subset of tokens which do not exist in any of the domain-specific lexicons, wherein each token in said second subset is assigned a sentiment score equal to zero, and 
 
 automatically generate a new lexicon in said specified domain comprising at least some of said first and second subsets. 
 
 
     
     
       23. The system of  claim 22 , wherein said program instructions are further executable to identify a third subset of tokens which exist in one or more of the domain-specific lexicons and one or more of the non-domain specific lexicons, wherein each of said tokens in said third subset is assigned a sentiment score equal to a linear interpolation of said associated sentiment score of said token in each of said domain-specific and non-domain specific lexicons; and wherein said new lexicon further comprises at least some of said third subset. 
     
     
       24. The system of  claim 22 , wherein said program instructions are further executable to:
 train a sentiment analysis classifier on a training set comprising at least said new lexicon; and 
 apply said sentiment analysis classifier to a text corpus. 
 
     
     
       25. The system of  claim 22 , wherein each of said associated sentiment orientations is selected from the group consisting of positive, negative, and neutral, and wherein said program instructions are further executable to automatically prune each of said tokens having a greater probability of being associated with (i) said neutral sentiment orientation, than with (ii) said positive and said negative sentiment orientations. 
     
     
       26. The system of  claim 22 , wherein said interpolation comprises weighting of each of said tokens based, at least in part, on a weighted average, wherein the weights in the weighted average are assigned based on a number of occurrences of each of said tokens in each of said domain-specific and non-domain specific lexicons. 
     
     
       27. The system of  claim 22 , wherein said interpolation comprises weighting each said token based, at least in part, on a weighted average, wherein the weights in the weighted average are assigned based on a confidence score of each of said tokens. 
     
     
       28. A computer program product comprising a non-transitory computer-readable storage medium having program instructions embodied therewith, the program instructions executable by at least one hardware processor to:
 receive one or more domain-specific lexicons in a specified domain, wherein each of said domain-specific lexicons comprises a plurality of tokens having each an associated sentiment orientation and sentiment score; 
 receive one or more non-domain specific lexicons, wherein each of said non-domain specific lexicons comprises a plurality of tokens having each an associated sentiment orientation and sentiment score; 
 automatically identify, in all said domain-specific and non-domain specific lexicons:
 (i) a first subset of tokens which exist in one or more of the domain-specific lexicons, wherein each of said tokens in said first subset is assigned a sentiment score equal to a linear interpolation of said associated sentiment score of said token in each of said domain-specific lexicons, and 
 (ii) a second subset of tokens which do not exist in any of the domain-specific lexicons, wherein each token in said second subset is assigned a sentiment score equal to zero; and 
 
 automatically generate a new lexicon in said specified domain comprising at least some of said first and second subsets. 
 
     
     
       29. The computer program product of  claim 28 , wherein said program instructions are further executable to identify a third subset of tokens which exist in one or more of the domain-specific lexicons and one or more of the non-domain specific lexicons, wherein each of said tokens in said third subset is assigned a sentiment score equal to a linear interpolation of said associated sentiment score of said token in each of said domain-specific and non-domain specific lexicons; and wherein said new lexicon further comprises at least some of said third subset. 
     
     
       30. The computer program product of  claim 28 , wherein said program instructions are further executable to:
 train a sentiment analysis classifier on a training set comprising at least said new lexicon; and 
 apply said sentiment analysis classifier to a text corpus. 
 
     
     
       31. The computer program product of  claim 28 , wherein each of said associated sentiment orientations is selected from the group consisting of positive, negative, and neutral, and wherein said program instructions are further executable to automatically prune each of said tokens having a greater probability of being associated with (i) said neutral sentiment orientation, than with (ii) said positive and said negative sentiment orientations. 
     
     
       32. The computer program product of  claim 28 , wherein said interpolation comprises weighting of each of said tokens based, at least in part, on a weighted average, wherein the weights in the weighted average are assigned based on a number of occurrences of each of said tokens in each of said domain-specific and non-domain specific lexicons. 
     
     
       33. The computer program product of  claim 28 , wherein said interpolation comprises weighting each said token based, at least in part, on a weighted average, wherein the weights in the weighted average are assigned based on a confidence score of each of said tokens.