System and method for eliciting open-ended natural language responses to questions to train natural language processors

Abstract

Systems and methods gathering text commands in response to a command context using a first crowdsourced are discussed herein. A command context for a natural language processing system may be identified, where the command context is associated with a command context condition to provide commands to the natural language processing system. One or more command creators associated with one or more command creation devices may be selected. A first application one the one or more command creation devices may be configured to display command creation instructions for each of the one or more command creators to provide text commands that satisfy the command context, and to display a field for capturing a user-generated text entry to satisfy the command creation condition in accordance with the command creation instructions. Systems and methods for reviewing the text commands using second and crowdsourced jobs are also presented herein.

What is claimed is: 
     
         1 . A computer-implemented method, the method being implemented in a computer system having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer system to perform the method, the method comprising:
 identifying a command context for a natural language processing system, the command context associated with a command context condition to provide commands to the natural language processing system;   selecting one or more command creators, the one or more command creators being associated with one or more command creation devices; and   configuring a first application on the one or more command creation devices to display the command context, to display command creation instructions for each of the one or more command creators to provide text commands that satisfy the command context, and to display a field for capturing a user-generated text entry to satisfy the command creation condition in accordance with the command creation instructions.   
     
     
         2 . The method of  claim 1 , wherein the first application is configured to display a plurality of command contexts, and to capture, for each of the plurality of command contexts, a plurality of user-generated text entries. 
     
     
         3 . The method of  claim 1 , wherein configuring the first application comprises periodically displaying a non-machine readable caption on the first application. 
     
     
         4 . The method of  claim 1 , wherein the non-machine readable caption comprises one or more of a non-machine readable image and a non-machine readable sound bite. 
     
     
         5 . The method of  claim 1 , further comprising:
 selecting one or more command reviewers, the one or more command reviewers being associated with one or more command reviewer devices; and   configuring a second application on the one or more command reviewer devices with command review instructions for the one or more command reviewers to review the user-generated text entry.   
     
     
         6 . The method of  claim 5 , wherein the command review instructions instruct the one or more command reviewers to review the syntax of the user-generated text entry. 
     
     
         7 . The method of  claim 5 , wherein the command review instructions instruct the one or more command reviewers to compare the user-generated text entry with model text language variants for the command context. 
     
     
         8 . The method of  claim 7 , wherein the second application is configured to display the user-generated text entry and the command context alongside the model text language variants, and to receive a selection from the one or more command reviewers whether the user-generated text entry is as similar to command context condition as one of the model text language variants. 
     
     
         9 . The method of  claim 1 , further comprising checking syntax of the user-generated text entry. 
     
     
         10 . The method of  claim 1 , further comprising storing the user-generated text entry in a transcription library. 
     
     
         11 . The method of  claim 1 , further comprising using the user-generated text entry for transcription of voice data during an end-user deployment phase of the natural language processing system. 
     
     
         12 . A system comprising:
 memory;   one or more physical processors programmed with one or more computer program instructions which, when executed, cause the one or more physical processors to:
 identify a command context for a natural language processing system, the command context associated with a command context condition to provide commands to the natural language processing system; 
 select one or more command creators, the one or more command creators being associated with one or more command creation devices; and 
 configure a first application on the one or more command creation devices to display the command context, to display command creation instructions for each of the one or more command creators to provide text commands that satisfy the command context, and to display a field for capturing a user-generated text entry to satisfy the command creation condition in accordance with the command creation instructions. 
   
     
     
         13 . The system of  claim 12 , wherein the first application is configured to display a plurality of command contexts, and to capture, for each of the plurality of command contexts, a plurality of user-generated text entries. 
     
     
         14 . The system of  claim 12 , wherein configuring the first application comprises periodically displaying a non-machine readable caption on the first application. 
     
     
         15 . The system of  claim 12 , wherein the non-machine readable caption comprises one or more of a non-machine readable image and a non-machine readable sound bite. 
     
     
         16 . The system of  claim 12 , wherein the instructions which, when executed, further cause the one or more physical processors to:
 select one or more command reviewers, the one or more command reviewers being associated with one or more command reviewer devices; and   configure a second application on the one or more command reviewer devices with command review instructions for the one or more command reviewers to review the user-generated text entry.   
     
     
         17 . The system of  claim 16 , wherein the command review instructions instruct the one or more command reviewers to review the syntax of the user-generated text entry. 
     
     
         18 . The system of  claim 16 , wherein the command review instructions instruct the one or more command reviewers to compare the user-generated text entry with model text language variants for the command context. 
     
     
         19 . The system of  claim 18 , wherein the second application is configured to display the user-generated text entry and the command context alongside the model text language variants, and to receive a selection from the one or more command reviewers whether the user-generated text entry is as similar to command context condition as one of the model text language variants. 
     
     
         20 . The system of  claim 12 , further comprising checking syntax of the user-generated text entry. 
     
     
         21 . The system of  claim 12 , further comprising storing the user-generated text entry in a transcription library. 
     
     
         22 . The system of  claim 12 , further comprising using the user-generated text entry for transcription of voice data during an end-user deployment phase of the natural language processing system. 
     
     
         23 . A computer program product comprising:
 one or more tangible, non-transitory computer-readable storage devices;   program instructions, stored on at least one of the one or more tangible, non-transitory computer-readable tangible storage devices that, when executed, cause a computer to:
 identify a command context for a natural language processing system, the command context associated with a command context condition to provide commands to the natural language processing system; 
 select one or more command creators, the one or more command creators being associated with one or more command creation devices; and 
 configure a first application on the one or more command creation devices to display the command context, to display command creation instructions for each of the one or more command creators to provide text commands that satisfy the command context, and to display a field for capturing a user-generated text entry to satisfy the command creation condition in accordance with the command creation instructions.