Method and system for generating natural language training data

Abstract

Provided is a system, method and computer-readable medium for generating data that may be used to train models for a natural language processing application. A system architect creates a plurality of sentence patterns that include entity variables and initiates sentence generation. Each entity is associated with one or more entity data sources. A language generator accepts the sentence patterns as inputs, and references the various entity sources to create a plurality of generated sentences. The generated sentences may be associated with a particular class and therefore used to train one or more statistical classification models and entity extraction models for associated models. The sentence generated process may be initiated and controlled using a user interface displayable on a computing device, the user interface in communication with the language generator module.

1 . A computer-implemented method for generating training data for training one or more models implemented in a natural language processing system, the method comprising:
 storing one or more sentence patterns, wherein each sentence pattern comprises one or more entity variables, each entity variable comprising a definition to replace the entity variable with an instance of an entity matching the definition;   accessing one or more entity sources, wherein each entity source comprises one or more entities to replace the one or more entity variables in accordance with the respective definition;   generating a plurality of sentences by replacing respective entity variables in each sentence pattern with one or more entities in accordance with the definitions; and   providing the plurality of sentences to train the one or more models.   
     
     
         2 . The method of  claim 1  wherein the plurality of sentences is a maximum number of sentences that can be generated based on the one or more sentence patterns, the one or more entity variables, and the one or more entities available from the one or more entity sources. 
     
     
         3 . The method of  claim 1  wherein the plurality of sentences match one or more input queries of a single class. 
     
     
         4 . The method of  claim 1  further comprising using the plurality of sentences to train one or more models to be implemented in a natural language processing system. 
     
     
         5 . The method of  claim 1  wherein generating the plurality of sentences comprises randomly selecting a sentence pattern from the one or more sentence patterns and, for each entity variable in the sentence pattern, randomly selecting an entity from the one or more entities matching the definition of the entity variable. 
     
     
         6 . The method of  claim 1  further comprising storing the plurality of sentences. 
     
     
         7 . The method of  claim 1  comprising providing a user interface to receive input to, at least one of: identify the one or more entity sources to be accessed; and define a sentence pattern. 
     
     
         8 . The method of  claim 1  wherein the one or more models are configured to perform at least one of: classifying an input query into one class of a set of one or more classes, identifying the input query as a specific command, and extracting one or more entities from the input query. 
     
     
         9 . The method of  claim 1  further comprising:
 receiving a dataset of input queries comprising natural language queries; 
 performing clustering on the dataset to cluster the respective input queries; and 
 adding selected clusters of input queries to the plurality of sentences to train one or more models. 
 
     
     
         10 . A non-transitory computer readable medium for generating training data for training one or more models implemented in a natural language processing system, the computer-readable medium comprising instructions that, when executed, cause a computer to perform operations comprising:
 storing one or more sentence patterns, wherein each sentence pattern comprises one or more entity variables, each entity variable comprising a definition to replace the entity variable with an instance of an entity matching the definition;   accessing one or more entity sources, wherein each entity source comprises one or more entities to replace the one or more entity variables in accordance with the respective definition;   generating a plurality of sentences by replacing respective entity variables in each sentence pattern with one or more entities in accordance with the definitions; and   providing the plurality of sentences to train the one or more models.   
     
     
         11 . The non-transitory computer-readable medium of  claim 10  wherein the plurality of sentences is a maximum number of sentences that can be generated based on the one or more sentence patterns, the one or more entity variables, and the one or more entities available from the one or more entity sources. 
     
     
         12 . The non-transitory computer-readable medium of  claim 10  wherein the plurality of sentences match one or more input queries of a single class. 
     
     
         13 . The non-transitory computer-readable medium of  claim 10  wherein the operations further comprise using the plurality of sentences to train one or more models to be implemented in a natural language processing system. (Original) The non-transitory computer-readable medium of  claim 10  wherein generating the plurality of sentences comprises randomly selecting a sentence pattern from the one or more sentence patterns and, for each entity variable in the sentence pattern, randomly selecting an entity from the one or more entities defined by the entity variable. 
     
     
         15 . The non-transitory computer-readable medium of  claim 10  wherein the operations further comprise storing the plurality of sentences. 
     
     
         16 . The non-transitory computer-readable medium of  claim 10  wherein the operations further comprise providing a user interface to receive input to, at least one of: identify the one or more entity sources to be accessed; and define a sentence pattern for manually identifying the one or more entity sources to be accessed. 
     
     
         17 . The non-transitory computer-readable medium of  claim 10  wherein the one or more models are configured to perform at least one of: classifying an input query into one class of a set of one or more classes, identifying the input query as a specific command, and extracting one or more entities from the input query. 
     
     
         18 . The non-transitory computer-readable medium of  claim 10  wherein the operations further comprise:
 receiving a dataset of input queries comprising natural language queries; 
 performing clustering on the dataset to cluster the respective input queries; and 
 adding selected clusters of input queries to the plurality of sentences to train one or more models. 
 
     
     
         19 . A computer system for generating training data for training one or more models implemented in a natural language processing system, the system comprising:
 one or more processors;   a memory coupled to the one or more processors and storing instructions and data for configuring the computer system to:
 store one or more sentence patterns, wherein each sentence pattern comprises one or more entity variables, each entity variable comprising a definition to replace the entity variable with an instance of an entity matching the definition; 
 access one or more entity sources, wherein each entity source comprises one or more entities to replace the one or more entity variables in accordance with the respective definition; 
 generate a plurality of sentences by replacing respective entity variables in each sentence pattern with one or more entities in accordance with the definitions; and 
 provide the plurality of sentences to train the one or more models. 
   
     
     
         20 . The computer system of  claim 19  further configured to provide a user interface to receive input to, at least one of: identify the one or more entity sources to be accessed; and receive input to define a sentence pattern. 
     
     
         21 . (canceled)