Open input classifier with entailment

Abstract

A natural language processing combination classifier is disclosed, leveraging an entailment classifier and optionally at least one of a pattern matching classifier and a trained machine learning (ML) classifier. Each of the different types of classifiers can be used to identify different categories of matches. For example, a pattern matching (e.g., regular expression) classifier may identify exact matches, an entailment classifier can obtain context-specific classifications based on likelihood of entailment to comparison data, and the SML classifier can obtain potential matches based on large-scale supervised training. The comparison data for the entailment classifier can be generated from small datasets, and can be readily updated without the need for retraining any machine learning models. Different types of classifiers can be processed using different logic to provide a user with the most appropriate response given a circumstance and given the user's open input.

What is claimed is: 
     
       1. A system, comprising:
 one or more data processors; and 
 a non-transitory computer-readable storage medium containing instructions which, when executed on the one or more data processors, cause the one or more data processors to perform operations including:
 receiving an input phrase associated with a topic; 
 accessing entailment comparison data, wherein the entailment comparison data includes a plurality of example inputs associated with a plurality of topics, wherein the plurality of topics includes the topic; 
 generating classification data for the input phrase, wherein generating the classification data includes applying an entailment classifier to the input phrase using the entailment comparison data, the entailment classifier including a machine learning model that has been trained to generate entailment values between two phrases, wherein the generated classification data is indicative of the topic, wherein applying the entailment classifier to the input phrase using the entailment comparison data includes:
 generating, for each of the plurality of example inputs of the entailment comparison data, an entailment value indicative of whether the input phrase i) implies the respective example input; ii) contradicts the respective example input; or iii) is neutral with respect to the respective example input; 
 generating, for each entailment value, a confidence score; 
 identifying a set of one or more relevant example inputs from the plurality of example inputs based at least in part on the entailment values and the confidence scores; 
 identifying a set of one or more matching topics from the plurality of topics based at least in part on the set of one or more relevant examples inputs; and 
 generating entailment classification data based at least in part on the set of one or more matching topics; and 
 
 generating a response based at least in part on the generated classification data. 
 
 
     
     
       2. The system of  claim 1 , wherein the operations further comprise presenting an input phrase prompt, wherein receiving the input phrase is in response to presenting the input phrase prompt, and wherein presenting the input phrase prompt includes presenting an input field for free text. 
     
     
       3. The system of  claim 1 , wherein generating the classification data further includes applying the input phrase to at least one of a pattern matching classifier and a supervised machine learning (SML) classifier. 
     
     
       4. The system of  claim 3 , wherein generating the classification data includes:
 applying the pattern matching classifier to the input phrase to generate pattern matching classification data based on one or more preset patterns; and 
 determining that the pattern matching classification data is indicative that the input phrase does not match the one or more preset patterns; 
 wherein applying the entailment classifier to the input phrase occurs in response to determining that the pattern matching classification data is indicative that the input phrase does not match the one or more preset patterns, wherein the entailment classification data is indicative of the topic, and wherein the generated classification data is based at least in part on the entailment classification data. 
 
     
     
       5. The system of  claim 3 , wherein generating the classification data includes:
 generating at least one of pattern matching classification data and SML classification data using the pattern matching classifier and SML classifier, respectively; and 
 identifying the topic using the entailment classification data and the at least one of the pattern matching classification data and the SML classification data. 
 
     
     
       6. The system of  claim 5 , wherein identifying the topic using the entailment classification data and the at least one of the pattern matching classification data and the SML classification data includes:
 determining one or more possible topics using first data, wherein the first data is selected from the group consisting of the entailment classification data and the at least one of the pattern matching classification data and the SML classification data; and 
 selecting the topic from the one or more possible topics using second data, wherein the second data is selected from the group consisting of the entailment classification data and the at least one of the pattern matching classification data and the SML classification data, and wherein the second data is different from the first data. 
 
     
     
       7. The system of  claim 3 , wherein the SML classifier is a bidirectional encoder representations from transformers (BERT) classifier. 
     
     
       8. The system of  claim 1 , wherein generating the classification data further includes applying the input phrase to at least one of a pattern matching classifier and a supervised machine learning (SML) classifier to generate pattern matching classification data and SML classification data, respectively; and wherein accessing the entailment comparison data is based at least in part on at least one of the pattern matching classification data and the SML classification data. 
     
     
       9. The system of  claim 1 , wherein accessing the entailment comparison data includes selecting the entailment comparison data from a plurality of potential entailment comparison datasets, wherein each potential entailment comparison dataset is associated with a unique subject matter. 
     
     
       10. The system of  claim 1 , wherein each of the plurality of topics of the entailment comparison data is associated with psychological therapy, and wherein generating the response facilitates delivery of psychological therapy to a user. 
     
     
       11. The system of  claim 10 , wherein generating the classification data further includes applying the input phrase to a supervised machine learning (SML) classifier that has been trained using training data that is not associated with psychological therapy. 
     
     
       12. A computer-implemented method, comprising:
 executed on the one or more data processors, cause the one or more data processors to perform operations including:
 receiving an input phrase associated with a topic; 
 accessing entailment comparison data, wherein the entailment comparison data includes a plurality of example inputs associated with a plurality of topics, wherein the plurality of topics includes the topic; 
 generating classification data for the input phrase, wherein generating the classification data includes applying an entailment classifier to the input phrase using the entailment comparison data, the entailment classifier including a machine learning model that has been trained to generate entailment values between two phrases, wherein the generated classification data is indicative of the topic, wherein applying the entailment classifier to the input phrase using the entailment comparison data includes:
 generating, for each of the plurality of example inputs of the entailment comparison data, an entailment value indicative of whether the input phrase i) implies the respective example input; ii) contradicts the respective example input; or iii) is neutral with respect to the respective example input; 
 generating, for each entailment value, a confidence score; 
 identifying a set of one or more relevant example inputs from the plurality of example inputs based at least in part on the entailment values and the confidence scores; 
 identifying a set of one or more matching topics from the plurality of topics based at least in part on the set of one or more relevant examples inputs; and 
 
 generating entailment classification data based at least in part on the set of one or more matching topics; and 
 generating a response based at least in part on the generated classification data. 
 
 
     
     
       13. The method of  claim 12 , further comprising presenting an input phrase prompt, wherein receiving the input phrase is in response to presenting the input phrase prompt, and wherein presenting the input phrase prompt includes presenting an input field for free text. 
     
     
       14. The method of  claim 12 , wherein generating the classification data further includes applying the input phrase to at least one of a pattern matching classifier and a supervised machine learning (SML) classifier. 
     
     
       15. The method of  claim 14 , wherein generating the classification data includes:
 applying the pattern matching classifier to the input phrase to generate pattern matching classification data based on one or more preset patterns; and 
 determining that the pattern matching classification data is indicative that the input phrase does not match the one or more preset patterns; 
 wherein applying the entailment classifier to the input phrase occurs in response to determining that the pattern matching classification data is indicative that the input phrase does not match the one or more preset patterns, wherein the entailment classification data is indicative of the topic, and wherein the generated classification data is based at least in part on the entailment classification data. 
 
     
     
       16. The method of  claim 14 , wherein generating the classification data includes:
 generating at least one of pattern matching classification data and SML classification data using the pattern matching classifier and SML classifier, respectively; and 
 identifying the topic using the entailment classification data and the at least one of the pattern matching classification data and the SML classification data. 
 
     
     
       17. The method of  claim 16 , wherein identifying the topic using the entailment classification data and the at least one of the pattern matching classification data and the SML classification data includes:
 determining one or more possible topics using first data, wherein the first data is selected from the group consisting of the entailment classification data and the at least one of the pattern matching classification data and the SML classification data; and 
 selecting the topic from the one or more possible topics using second data, wherein the second data is selected from the group consisting of the entailment classification data and the at least one of the pattern matching classification data and the SML classification data, and wherein the second data is different from the first data. 
 
     
     
       18. The method of  claim 14 , wherein the SML classifier is a bidirectional encoder representations from transformers (BERT) classifier. 
     
     
       19. The method of  claim 12 , wherein generating the classification data further includes applying the input phrase to at least one of a pattern matching classifier and a supervised machine learning (SML) classifier to generate pattern matching classification data and SML classification data, respectively; and wherein accessing the entailment comparison data is based at least in part on at least one of the pattern matching classification data and the SML classification data. 
     
     
       20. The method of  claim 12 , wherein accessing the entailment comparison data includes selecting the entailment comparison data from a plurality of potential entailment comparison datasets, wherein each potential entailment comparison dataset is associated with a unique subject matter. 
     
     
       21. The method of  claim 12 , wherein each of the plurality of topics of the entailment comparison data is associated with psychological therapy, and wherein generating the response facilitates delivery of psychological therapy to a user. 
     
     
       22. The method of  claim 21 , wherein generating the classification data further includes applying the input phrase to a supervised machine learning (SML) classifier that has been trained using training data that is not associated with psychological therapy. 
     
     
       23. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause a data processing apparatus to perform operations including:
 receiving an input phrase associated with a topic; 
 accessing entailment comparison data, wherein the entailment comparison data includes a plurality of example inputs associated with a plurality of topics, wherein the plurality of topics includes the topic; 
 generating classification data for the input phrase, wherein generating the classification data includes applying an entailment classifier to the input phrase using the entailment comparison data, the entailment classifier including a machine learning model that has been trained to generate entailment values between two phrases, wherein the generated classification data is indicative of the topic, wherein applying the entailment classifier to the input phrase using the entailment comparison data includes:
 generating, for each of the plurality of example inputs of the entailment comparison data, an entailment value indicative of whether the input phrase i) implies the respective example input; ii) contradicts the respective example input; or iii) is neutral with respect to the respective example input; 
 generating, for each entailment value, a confidence score; 
 identifying a set of one or more relevant example inputs from the plurality of example inputs based at least in part on the entailment values and the confidence scores; 
 identifying a set of one or more matching topics from the plurality of topics based at least in part on the set of one or more relevant examples inputs; and 
 generating entailment classification data based at least in part on the set of one or more matching topics; and 
 
 generating a response based at least in part on the generated classification data. 
 
     
     
       24. The computer-program product of  claim 23 , wherein the operations further comprise presenting an input phrase prompt, wherein receiving the input phrase is in response to presenting the input phrase prompt, and wherein presenting the input phrase prompt includes presenting an input field for free text. 
     
     
       25. The computer-program product of  claim 23 , wherein generating the classification data further includes applying the input phrase to at least one of a pattern matching classifier and a supervised machine learning (SML) classifier. 
     
     
       26. The computer-program product of  claim 25 , wherein generating the classification data includes:
 applying the pattern matching classifier to the input phrase to generate pattern matching classification data based on one or more preset patterns; and 
 determining that the pattern matching classification data is indicative that the input phrase does not match the one or more preset patterns; 
 wherein applying the entailment classifier to the input phrase occurs in response to determining that the pattern matching classification data is indicative that the input phrase does not match the one or more preset patterns, wherein the entailment classification data is indicative of the topic, and wherein the generated classification data is based at least in part on the entailment classification data. 
 
     
     
       27. The computer-program product of  claim 25 , wherein generating the classification data includes:
 generating at least one of pattern matching classification data and SML classification data using the pattern matching classifier and SML classifier, respectively; and 
 identifying the topic using the entailment classification data and the at least one of the pattern matching classification data and the SML classification data. 
 
     
     
       28. The computer-program product of  claim 27 , wherein identifying the topic using the entailment classification data and the at least one of the pattern matching classification data and the SML classification data includes:
 determining one or more possible topics using first data, wherein the first data is selected from the group consisting of the entailment classification data and the at least one of the pattern matching classification data and the SML classification data; and 
 selecting the topic from the one or more possible topics using second data, wherein the second data is selected from the group consisting of the entailment classification data and the at least one of the pattern matching classification data and the SML classification data, and wherein the second data is different from the first data. 
 
     
     
       29. The computer-program product of  claim 23 , wherein generating the classification data further includes applying the input phrase to at least one of a pattern matching classifier and a supervised machine learning (SML) classifier to generate pattern matching classification data and SML classification data, respectively; and wherein accessing the entailment comparison data is based at least in part on at least one of the pattern matching classification data and the SML classification data. 
     
     
       30. The computer-program product of  claim 23 , wherein each of the plurality of topics of the entailment comparison data is associated with psychological therapy, wherein generating the response facilitates delivery of psychological therapy to a user, and wherein generating the classification data further includes applying the input phrase to a supervised machine learning (SML) classifier that has been trained using training data that is not associated with psychological therapy.