Supervised machine learning for automated assistants

Abstract

Disclosed are methods and systems for supervised machine learning for automated assistants. An example method includes: receiving an automated assistant transcript comprising a plurality of records, wherein each record of the plurality of records comprises a query, a classification of the query, an intent associated with the query, and a responsive action associated with the intent; receiving, via a graphical user interface (GUI), a user input indicating an approval of a new automated assistant transcript record; comparing the new automated assistant transcript record to one or more records of the plurality of records; and responsive to detecting a conflict of the new automated assistant transcript record with one or more records of the plurality of records, displaying, via the GUI, a notification of the conflict.

What is claimed is: 
     
         1 . A method, comprising:
 receiving, by a computer system, an automated assistant transcript comprising a plurality of records, wherein each record of the plurality of records comprises a query, a classification of the query, an intent associated with the query, and a responsive action associated with the intent;   comparing the new automated assistant transcript record to one or more records of the plurality of records;   responsive detecting a conflict of the new automated assistant transcript record with an existing record of the plurality of records, modifying one or more fields of the new automated assistant transcript record;   appending the new automated assistant transcript record to the automated assistant transcript; and   utilizing the automated assistant transcript for training a first set of classification models and a second set of classification models, wherein each classification model of the first set of classification models is employed to determine a degree of association of an input query with a topic of a predefined set of topics, and wherein each classification model of the second set of classification models is employed to determine a degree of association of the input query with an intent of a predefined set of intents associated with the specified topic.   
     
     
         2 . The method of  claim 1 , further comprising:
 responsive to failing to detect a conflict of the new automated assistant transcript record with the plurality of records, appending the new automated assistant transcript record to the automated assistant transcript.   
     
     
         3 . The method of  claim 1 , further comprising:
 validating the classification models; and   publishing the classification models to a model deployment environment.   
     
     
         4 . The method of  claim 3 , wherein validating a classification model further comprises:
 running the classification model on a plurality of data items of a validation data set;   evaluating a quality metric reflecting a difference between an actual output of the classification model and a desired output of the classification model; and   determining whether the quality metric value falls within a predetermined range.   
     
     
         5 . The method of  claim 1 , wherein comparing the new automated assistant transcript record to one or more records further comprises:
 computing values of one or more classification features associated with the new automated assistant transcript record.   
     
     
         6 . The method of  claim 1 , wherein the query is represented by a vector of classification features, wherein each element of the vector represents a number of occurrences in the query of a word identified by an index of the element. 
     
     
         7 . The method of  claim 1 , wherein the query is associated with a parameter identifying an object of an action identified by the intent. 
     
     
         8 . A system, comprising:
 a memory; and   a processing device operatively coupled to the memory, wherein the processing device is configured to:
 receive an automated assistant transcript comprising a plurality of records, wherein each record of the plurality of records comprises a query, a classification of the query, an intent associated with the query, and a responsive action associated with the intent; 
 compare the new automated assistant transcript record to one or more records of the plurality of records; 
 responsive detecting a conflict of the new automated assistant transcript record with an existing record of the plurality of records, modify one or more fields of the new automated assistant transcript record; 
 append the new automated assistant transcript record to the automated assistant transcript; and 
 utilize the automated assistant transcript for training a first set of classification models and a second set of classification models, wherein each classification model of the first set of classification models is employed to determine a degree of association of an input query with a topic of a predefined set of topics, and wherein each classification model of the second set of classification models is employed to determine a degree of association of the input query with an intent of a predefined set of intents associated with the specified topic. 
   
     
     
         9 . The system of  claim 8 , wherein the processing device is further configured to:
 responsive to failing to detect a conflict of the new automated assistant transcript record with the plurality of records, append the new automated assistant transcript record to the automated assistant transcript.   
     
     
         10 . The system of  claim 8 , wherein the processing device is configured to:
 validate the classification models; and   publish the classification models to a model deployment environment.   
     
     
         11 . The system of  claim 10 , wherein validating a classification model further comprises:
 running the classification model on a plurality of data items of a validation data set;   evaluating a quality metric reflecting a difference between an actual output of the classification model and a desired output of the classification model; and   determining whether the quality metric value falls within a predetermined range.   
     
     
         12 . The system of  claim 8 , wherein comparing the new automated assistant transcript record to one or more records further comprises:
 computing values of one or more classification features associated with the new automated assistant transcript record.   
     
     
         13 . The system of  claim 8 , wherein the query is represented by a vector of classification features, wherein each element of the vector represents a number of occurrences in the query of a word identified by an index of the element. 
     
     
         14 . The system of  claim 8 , wherein the query is associated with a parameter identifying an object of an action identified by the intent. 
     
     
         15 . A non-transitory computer-readable storage medium comprising executable instructions which, when executed by a computer system, cause the computer system to:
 receive an automated assistant transcript comprising a plurality of records, wherein each record of the plurality of records comprises a query, a classification of the query, an intent associated with the query, and a responsive action associated with the intent;   compare the new automated assistant transcript record to one or more records of the plurality of records;   responsive detecting a conflict of the new automated assistant transcript record with an existing record of the plurality of records, modify one or more fields of the new automated assistant transcript record;   append the new automated assistant transcript record to the automated assistant transcript; and   utilize the automated assistant transcript for training a first set of classification models and a second set of classification models, wherein each classification model of the first set of classification models is employed to determine a degree of association of an input query with a topic of a predefined set of topics, and wherein each classification model of the second set of classification models is employed to determine a degree of association of the input query with an intent of a predefined set of intents associated with the specified topic.   
     
     
         16 . The non-transitory computer-readable storage medium of  claim 15 , further comprising executable instructions which, when executed by the computer system, cause the computer system to:
 responsive to failing to detect a conflict of the new automated assistant transcript record with the plurality of records, append the new automated assistant transcript record to the automated assistant transcript.   
     
     
         17 . The non-transitory computer-readable storage medium of  claim 15 , further comprising executable instructions which, when executed by the computer system, cause the computer system to:
 validate the classification models; and   publish the classification models to a model deployment environment.   
     
     
         18 . The non-transitory computer-readable storage medium of  claim 17 , wherein validating a classification model further comprises:
 running the classification model on a plurality of data items of a validation data set;   evaluating a quality metric reflecting a difference between an actual output of the classification model and a desired output of the classification model; and   determining whether the quality metric value falls within a predetermined range.   
     
     
         19 . The non-transitory computer-readable storage medium of  claim 15 , wherein comparing the new automated assistant transcript record to one or more records further comprises:
 computing values of one or more classification features associated with the new automated assistant transcript record.   
     
     
         20 . The non-transitory computer-readable storage medium of  claim 15 , wherein the query is represented by a vector of classification features, wherein each element of the vector represents a number of occurrences in the query of a word identified by an index of the element.