Generating additional training data for a natural language understanding engine

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating additional training data for a natural language understanding engine. One of the methods includes: obtaining data identifying (i) a first input conversational turn and (ii) a first annotation, determining that the first annotation accurately characterized the first input conversational turn, determining that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn, in response to the determining, obtaining one or more first paraphrases of the first input conversational turn; and generating, for each of the one or more first paraphrases, a respective first training example that identifies the first annotation as the correct annotation for the first paraphrase; and training the natural language understanding engine on at least the first training examples.

1 . A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:
 obtaining, during operation of a computer-implemented dialogue system comprising a natural language understanding engine, data identifying (i) a first input conversational turn that was provided as input to the natural language understanding engine during a dialogue between a user and the computer-implemented dialogue system and (ii) a first annotation of the first input conversational turn generated by the natural language understanding engine, wherein the natural language understanding engine has been trained on a first set of training data comprising a plurality of training conversational turns;   determining that the first annotation did not accurately characterize the first input conversational turn;   in response to determining that the first annotation did not accurately characterize the first input conversational turn:
 determining a correct annotation for the first conversational turn; 
 obtaining one or more first paraphrases of the first input conversational turn; and 
 generating, for each of the one or more first paraphrases, a respective first training example that identifies the correct annotation for the first conversational turn as the correct annotation for the first paraphrase; and 
   
       training the natural language understanding engine on at least the first training examples. 
     
     
         2 . The system of  claim 1 , the operations further comprising obtaining a confidence score generated by the natural language understanding engine that represents a confidence that the first annotation is an accurate characterization of the first input conversational turn, and wherein determining that the first annotation did not accurately characterize the first input conversational turn comprises determining that the confidence score fails to exceed a threshold. 
     
     
         3 . The system of  claim 1 , wherein determining that the first annotation did not accurately characterize the first input conversational turn comprises:
 processing the first input conversational turn and the first annotation using a post-hoc annotation machine learning model that is configured to generate as output a quality score that represents a likelihood that the first annotation is an accurate characterization of the first input conversational turn; and   determining that the quality score fails to exceed a threshold.   
     
     
         4 . The system of  claim 3 , wherein the post-hoc annotation machine learning model is configured to receive as input (i) the first input conversational turn, (ii) the first annotation, (iii) one or more conversational turns occurring before the first input conversational turn in the dialogue, and (iv) one or more conversational turns occurring after the first input conversational turn in the dialogue. 
     
     
         5 . The system of  claim 1 , wherein determining that the first annotation does not accurately characterize the first input conversational turn comprises:
 determining that conversational turns occurring after the first input conversational turn in the dialogue indicate that the first annotation did not accurately characterize the first input conversational turn.   
     
     
         6 . The system of  claim 5 , wherein determining that conversational turns occurring after the first input conversational turn in the dialogue indicate that the first annotation did not accurately characterize the first input conversational turn comprises:
 determining (i) that a task was completed as a result of the dialogue and (ii) that one or more slot values assigned in the first annotation were changed or removed by the user in one or more of the conversational turns occurring after the first input conversational turn in the dialogue.   
     
     
         7 . A method comprising:
 obtaining, during operation of a computer-implemented dialogue system comprising a natural language understanding engine, data identifying (i) a first input conversational turn that was provided as input to the natural language understanding engine during a dialogue between a user and the computer-implemented dialogue system and (ii) a first annotation of the first input conversational turn generated by the natural language understanding engine, wherein the natural language understanding engine has been trained on a first set of training data comprising a plurality of training conversational turns;   determining that the first annotation did not accurately characterize the first input conversational turn;   in response to determining that the first annotation did not accurately characterize the first input conversational turn:
 determining a correct annotation for the first conversational turn; 
 obtaining one or more first paraphrases of the first input conversational turn; and 
 generating, for each of the one or more first paraphrases, a respective first training example that identifies the correct annotation for the first conversational turn as the correct annotation for the first paraphrase; and 
   training the natural language understanding engine on at least the first training examples.   
     
     
         8 . The method of  claim 7 , further comprising obtaining a confidence score generated by the natural language understanding engine that represents a confidence that the first annotation is an accurate characterization of the first input conversational turn, and wherein determining that the first annotation does not accurately characterize the first input conversational turn comprises determining that the confidence score fails to exceed a threshold. 
     
     
         9 . The method of  claim 7 , wherein determining that the first annotation accurately characterized the first input conversational turn comprises:
 processing the first input conversational turn and the first annotation using a post-hoc annotation machine learning model that is configured to generate as output a quality score that represents a likelihood that the first annotation is an accurate characterization of the first input conversational turn; and   determining that the quality score fails to exceed a threshold.   
     
     
         10 . The method of  claim 9 , wherein the post-hoc annotation machine learning model is configured to receive as input (i) the first input conversational turn, (ii) the first annotation, (iii) one or more conversational turns occurring before the first input conversational turn in the dialogue, and (iv) one or more conversational turns occurring after the first input conversational turn in the dialogue. 
     
     
         11 . The method of  claim 7 , wherein determining that the first annotation does not accurately characterize the first input conversational turn comprises:
 determining that conversational turns occurring after the first input conversational turn in the dialogue indicate that the first annotation does not accurately characterize the first input conversational turn.   
     
     
         12 . The method of  claim 11 , wherein determining that conversational turns occurring after the first input conversational turn in the dialogue indicate that the first annotation does not accurately characterize the first input conversational turn comprises:
 determining (i) that a task was completed as a result of the dialogue and (ii) that one or more slot values assigned in the first annotation were changed or removed by the user in one or more of the conversational turns occurring after the first input conversational turn in the dialogue.   
     
     
         13 . One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:
 obtaining, during operation of a computer-implemented dialogue system comprising a natural language understanding engine, data identifying (i) a first input conversational turn that was provided as input to the natural language understanding engine during a dialogue between a user and the computer-implemented dialogue system and (ii) a first annotation of the first input conversational turn generated by the natural language understanding engine, wherein the natural language understanding engine has been trained on a first set of training data comprising a plurality of training conversational turns;   determining that the first annotation does not accurately characterize the first input conversational turn;   in response to determining that the first annotation did not accurately characterize the first input conversational turn:
 determining a correct annotation for the first conversational turn; 
 obtaining one or more first paraphrases of the first input conversational turn; and 
 generating, for each of the one or more first paraphrases, a respective first training example that identifies the correct annotation for the first conversational turn as the correct annotation for the first paraphrase; and 
   training the natural language understanding engine on at least the first training examples.   
     
     
         14 . The computer-readable storage media of  claim 13 , further comprising obtaining a confidence score generated by the natural language understanding engine that represents a confidence that the first annotation is an accurate characterization of the first input conversational turn, and wherein determining that the first annotation does not accurately characterize the first input conversational turn comprises determining that the confidence score fails to exceed a threshold. 
     
     
         15 . The computer-readable storage media of  claim 13 , wherein determining that the first annotation accurately characterized the first input conversational turn comprises:
 processing the first input conversational turn and the first annotation using a post-hoc annotation machine learning model that is configured to generate as output a quality score that represents a likelihood that the first annotation is an accurate characterization of the first input conversational turn; and   determining that the quality score fails to exceed a threshold.   
     
     
         16 . The computer-readable storage media of  claim 15 , wherein the post-hoc annotation machine learning model is configured to receive as input (i) the first input conversational turn, (ii) the first annotation, (iii) one or more conversational turns occurring before the first input conversational turn in the dialogue, and (iv) one or more conversational turns occurring after the first input conversational turn in the dialogue. 
     
     
         17 . The computer-readable storage media of  claim 13 , wherein determining that the first annotation does not accurately characterize the first input conversational turn comprises:
 determining that conversational turns occurring after the first input conversational turn in the dialogue indicate that the first annotation does not accurately characterize the first input conversational turn.   
     
     
         18 . The computer-readable storage media of  claim 17 , wherein determining that conversational turns occurring after the first input conversational turn in the dialogue indicate that the first annotation does not accurately characterize the first input conversational turn comprises:
 determining (i) that a task was completed as a result of the dialogue and (ii) that one or more slot values assigned in the first annotation were changed or removed by the user in one or more of the conversational turns occurring after the first input conversational turn in the dialogue.