Systems and methods for constructing an artificially diverse corpus of training data samples for training a contextually-biased model for a machine learning-based dialogue system

Abstract

Systems and methods for constructing an artificially diverse corpus of training data includes evaluating a corpus of utterance-based training data samples, identifying a slot replacement candidate; deriving distinct skeleton utterances that include the slot replacement candidate, wherein deriving the distinct skeleton utterances includes replacing slots of each of the plurality of distinct utterance training samples with one of a special token and proper slot classification labels; selecting a subset of the distinct skeleton utterances; converting each of the distinct skeleton utterances of the subset back to distinct utterance training samples while still maintaining the special token at a position of the slot replacement candidate; altering a percentage of the distinct utterance training samples with a distinct randomly-generated slot token value at the position of the slot replacement candidate; and constructing the artificially diverse corpus of training samples based on a collection of the percentage of the distinct utterance training samples.

What is claimed is: 
     
         1 . A method for building an artificially diverse corpus of training data samples for training a contextually-biased model, the method comprising:
 identifying a slot replacement candidate from within a training corpus of distinct text-based training data samples, wherein the slot replacement candidate relates to a slot within a subset of the text-based training data samples of the training corpus that have one or more tokens that are evaluated for replacement with one or more randomly-generated slot token values;   deriving a plurality of distinct skeleton texts, wherein the deriving includes a transformation of a plurality of text-based training data samples having the slot replacement candidate of the training corpus to the one or more distinct skeleton texts by replacing the slot replacement candidate within each of the plurality of text-based training data samples with a special token;   identifying the slot replacement candidate as one of a safe slot for replacement or an unsafe slot for replacement with the one or more randomly-generated slot token values based on an evaluation of the plurality of distinct skeleton texts,   wherein if the slot replacement candidate is identified as the safe slot for replacement:
 (i) reverting the plurality of distinct skeleton texts to a set of a plurality of text-based training data samples within the training corpus; 
 (ii) substituting the slot replacement candidate with a distinct randomly-generated slot token value in a percentage of the set of plurality of text-based training data samples; and 
   training one or more machine learning models using the training corpus of text-based training data samples,   wherein, once trained, the one or more machine learning models predict a classification of one or more slots of a text-based input to a machine learning-based dialogue system that generates a response based on the classification of the one or more slots of the text-based input.   
     
     
         2 . The method according to  claim 1 , wherein
 reverting the plurality of distinct skeleton texts includes maintaining the special token at a position of the slot replacement candidate in each of the plurality of text-based training data samples within the set.   
     
     
         3 . The method according to  claim 1 , wherein
 the deriving further includes replacing one or more slots distinct from the slot replacement candidate within each of the plurality of text-based training data samples with a slot classification label.   
     
     
         4 . The method according to  claim 1 , wherein
 identifying the slot replacement candidate includes:
 selecting a target slot of the one or more text-based training data samples of the training corpus as the slot replacement candidate if an enumerability of the target slot to different token values satisfies or exceeds an enumeration threshold, wherein the enumeration threshold comprises a minimum number of possible token values for the target slot. 
   
     
     
         5 . The method according to  claim 1 , wherein
 the safe slot relates to a target slot that is suitable for replacement with the randomly-generated token value, and wherein the unsafe slot relates to a distinct target slot that is not suitable for replacement with a distinct randomly-generated token value.   
     
     
         6 . The method according to  claim 1 , wherein
 identifying the slot replacement candidate as one of the safe slot or the unsafe slot includes:
 measuring an average token length for the slot replacement candidate within the training corpus; 
 evaluating the average token length of the slot replacement candidate against a token length threshold; 
 if the average token length of the slot replacement candidate satisfies or exceeds the token length threshold, categorizing the slot replacement as the unsafe slot. 
   
     
     
         7 . The method according to  claim 1 , wherein
 identifying the slot replacement candidate as one of the safe slot or the unsafe slot includes:
 measuring an average token length for the slot replacement candidate within the training corpus; 
 evaluating the average token length of the slot replacement candidate against a token length threshold; 
 if the average token length of the slot replacement candidate does not exceed the token length threshold, categorizing the slot replacement as the safe slot. 
   
     
     
         8 . The method according to  claim 1 , wherein
 identifying the slot replacement candidate as one of the safe slot or the unsafe slot includes:
 measuring an enumerability attribute for the slot replacement candidate within the training corpus; 
 evaluating the enumerability attribute of the slot replacement candidate against an enumeration threshold; 
 if the enumerability attribute of the slot replacement candidate satisfies or exceeds the enumeration threshold, categorizing the slot replacement candidate as the safe slot. 
   
     
     
         9 . The method according to  claim 1 , wherein
 identifying the slot replacement candidate as one of the safe slot or the unsafe slot includes:
 measuring an enumerability attribute for the slot replacement candidate within the training corpus; 
 evaluating the enumerability attribute of the slot replacement candidate against an enumeration threshold; 
 if the enumerability attribute of the slot replacement candidate does not exceed the enumeration threshold, categorizing the slot replacement candidate as the unsafe slot. 
   
     
     
         10 . The method according to  claim 1 , wherein
 deriving the one or more distinct skeleton texts for the slot replacement candidate includes:
 creating a skeleton set for containing each of a plurality of distinct potential skeleton texts for the slot replacement candidate; 
 wherein transforming each of the plurality of text-based training data samples includes:
 replacing one or more tokens at a position of the slot replacement candidate with the special token indicating a focus of a potential skeleton text; and 
 if one or more other slots exist within each of the plurality of text-based training data samples, replacing one or more tokens at one or more positions of the one or more other slots with one or more proper slot classification labels; 
 
 if a duplicate skeleton text does not exist in the skeleton set, adding each of the plurality of text-based training data samples, as converted, as one of the plurality of distinct potential skeleton texts. 
   
     
     
         11 . The method according to  claim 10 , wherein
 if a difference between a first potential skeleton text of the plurality of distinct potential skeleton texts and a second potential skeleton text of the plurality of distinct potential skeleton texts within the skeleton set is only by an one-edit substitution with a slot classification label, designating both the first and the second potential skeleton texts as unsafe for modification of the slot replacement candidate with the randomly-generated slot token value.   
     
     
         12 . The method according to  claim 10 , further comprising:
 extracting from the skeleton set a subset of the plurality of distinct potential skeleton texts that are designated as safe for modification of the slot replacement candidate with the randomly-generated slot token value, wherein the subset comprises the one or more distinct skeleton texts.   
     
     
         13 . The method according to  claim 1 , wherein
 reverting the plurality of distinct skeleton texts to a set of a plurality of text-based training data samples within the training corpus includes:
 retrieving original slot token values for each slot of the one or more distinct skeleton texts having one or more slot classification labels in the place of original slot token values; and 
 replacing the one or more slot classification labels with one of the original slot token values. 
   
     
     
         14 . The method according to  claim 1 , wherein
 if a target slot of an original text corresponding to the slot replacement candidate includes multiple tokens, the transformation includes replacing the special token of a given distinct skeleton text with multiple distinct randomly-generated token values matching a number of the multiple tokens.   
     
     
         15 . The method according to  claim 1 , further comprising:
 constructing an artificially diverse corpus of training data samples for training the one or more machine learning model includes:
 artificially expanding the artificially diverse corpus by augmenting the training corpus of text-based training data samples with the percentage of the set of the plurality of text-based training data samples. 
   
     
     
         16 . The method according to  claim 1 , further comprising:
 constructing an artificially diverse corpus of training data samples for training the contextually-biased model includes:
 artificially diversifying the artificially diverse corpus by replacing original text-based training data samples of the training corpus of text-based training data samples with the percentage of the set of the plurality of text-based training data samples. 
   
     
     
         17 . The method according to  claim 1 , wherein
 the randomly-generated token value comprises one or more nonsensical terms or set of characters that do not confer any real-world or computer-based meaning within the training corpus.   
     
     
         18 . A system for building an artificially diverse corpus of training data samples for training a contextually-biased model, the system comprising:
 a machine learning-based automated dialogue service implemented by one or more hardware computing servers that:
 identify a slot replacement candidate from within a training corpus of distinct text-based training data samples, wherein the slot replacement candidate relates to a slot within a subset of the text-based training data samples of the training corpus that have one or more tokens that are evaluated for replacement with one or more randomly-generated slot token values; 
 derive a plurality of distinct skeleton texts, wherein the deriving includes a transformation of a plurality of text-based training data samples having the slot replacement candidate of the training corpus to the one or more distinct skeleton texts by replacing the slot replacement candidate within each of the plurality of text-based training data samples with a special token; 
 identify the slot replacement candidate as one of a safe slot for replacement or an unsafe slot for replacement with the one or more randomly-generated slot token values based on an evaluation of the plurality of distinct skeleton texts, 
 wherein if the slot replacement candidate is identified as the safe slot for replacement:
 (i) reverting the plurality of distinct skeleton texts to a set of a plurality of text-based training data samples within the training corpus; 
 (ii) substituting the slot replacement candidate with a distinct randomly-generated slot token value in a percentage of the set of plurality of text-based training data samples; and 
 
 train one or more machine learning models using the training corpus of text-based training data samples, 
 wherein, once trained, the one or more machine learning models predict a classification of one or more slots of a text-based input to a machine learning-based dialogue system that generates a response based on the classification of the one or more slots of the text-based input. 
   
     
     
         19 . The system according to  claim 18 , wherein
 wherein the safe slot relates to a target slot that is suitable for replacement with the randomly-generated token value, and wherein the unsafe slot relates to a distinct target slot that is not suitable for replacement with a distinct randomly-generated token value.   
     
     
         20 . A method for building a corpus of training data samples for training a machine learning model, the method comprising:
 identifying a slot replacement candidate from within a training corpus of distinct text-based training data samples, wherein the slot replacement candidate relates to a slot within a subset of the text-based training data samples of the training corpus that have one or more tokens that are evaluated for replacement with one or more randomly-generated slot token values;   deriving a plurality of distinct skeleton texts, wherein the deriving includes a transformation of a plurality of text-based training data samples having the slot replacement candidate of the training corpus to the one or more distinct skeleton texts by replacing the slot replacement candidate within each of the plurality of text-based training data samples with a special token;   identifying the slot replacement candidate as one of a safe slot for replacement or an unsafe slot for replacement with the one or more randomly-generated slot token values based on an evaluation of the plurality of distinct skeleton texts,   wherein if the slot replacement candidate is identified as the safe slot for replacement, substituting the slot replacement candidate with a distinct randomly-generated slot token value in a percentage of the plurality of distinct skeleton texts; and   training one or more machine learning models using the training corpus of text-based training data samples,
 wherein, once trained, the one or more machine learning models predict a classification of one or more slots of a text-based input to a machine learning-based dialogue system that generates a response based on the classification of the one or more slots of the text-based input.