System and method for guided synthesis of training data

Abstract

Embodiments described herein provide mechanisms to generate synthetic data that meets customized invariance and diversity criteria at scale using a combination of machine and human activities. Embodiments can be used, for example, to generate a large set of labeled data that preserves invariances that are relevant to a specified target application (e.g., training a ML model to produce a given inference) while expanding both the quantity and diversity/variation of data relevant to that application. In some embodiments, a complex workflow can be defined that combines stages having both machine processes and human processes that provide guidance from assessors to generators such that subsequent data generation is improved through feedback that can include human input at scale.

What is claimed is: 
     
         1 . A computer-implemented method for guided synthesis of data, comprising:
 receiving a set of input data from a data store;   transforming the set of input data, by one or more generators, to generate a set of output data;   storing the generated set of output data;   producing an assessment, by one or more assessors, of the set of output data against a set of characteristics to determine whether the set of characteristics are met by the set of output data, wherein the one or more generators and one or more assessors are used in a configurable series, parallel, or hierarchical workflow; and   storing the generated set of synthetic data in a second data store.   
     
     
         2 . The computer-implemented method of  claim 1 , wherein one or more of the generators comprises a machine learning generator. 
     
     
         3 . The computer-implemented method of  claim 1 , wherein one or more of the generators comprises a human process, wherein the human process comprises computer code configured to perform operations and support interactions with a human specialist. 
     
     
         4 . The computer-implemented method of  claim 1 , wherein the assessment is produced by a machine learning assessor. 
     
     
         5 . The computer-implemented method of  claim 1 , wherein the assessment is produced using a human process, wherein the human process comprises computer code configured to perform operations and support interactions with a human specialist. 
     
     
         6 . The computer-implemented method of  claim 1 , wherein the set of computer-executable instructions further comprises instructions for:
 specifying a combination of stages of a workflow, the stages of the workflow including a generator stage and an assessor stage;   configuring inputs and outputs of the stages of the workflow; and   configuring connections between the stages of the workflow.   
     
     
         7 . The computer-implemented method of  claim 6 , wherein the specified stages of a workflow include a combination of machine learning processes and human processes. 
     
     
         8 . The computer-implemented method of  claim 1 , wherein the set of computer-executable instructions further comprises instructions for augmenting the set of output data based on the assessment to generate a set of synthetic data. 
     
     
         9 . The computer-implemented method of  claim 1 , wherein the set of computer-executable instructions further comprises instructions for:
 packaging, by a dispatcher service, a generated set of data as a task for presentation to a human specialist using a task user interface template; and   receiving a task result, by the dispatcher service, from the human specialist and returning the task result to the human process.   
     
     
         10 . The computer-implemented method of  claim 9 , further comprising validating, by the dispatcher service, the task result. 
     
     
         11 . The computer-implemented method of  claim 9 , further comprising:
 providing defined groups of human specialists; and   specifying one of the defined groups of human specialists in the task.   
     
     
         12 . The computer-implemented method of  claim 1 , further comprising:
 monitoring, by a gate process, stored sets of generated synthetic data in the second data store; and   responsive to determining that a threshold has been reached regarding the generated set of synthetic data, generating a trigger signal for stopping the assessment of output data.   
     
     
         13 . The computer-implemented method of  claim 12 , further comprising responsive to determining that a threshold has been reached regarding the generated set of synthetic data, generating a second trigger signal for causing a filter process to provide available synthetic data from the second data store. 
     
     
         14 . A computer program product comprising a non-transitory, computer-readable medium storing thereon a set of computer-executable instructions, the set of computer-executable instructions comprising instructions for:
 receiving a set of input data from a data store;   transforming the set of input data, by one or more generators, to generate a set of output data;   storing the generated set of output data;   producing an assessment, by one or more assessors, of the set of output data against a set of characteristics to determine whether the set of characteristics are met by the set of output data, wherein the one or more generators and one or more assessors are used in a configurable series, parallel, or hierarchical workflow; and   storing the generated set of synthetic data in a second data store.   
     
     
         15 . The computer program product of  claim 14 , wherein the one or more generators and the one or more assessors are comprised of a combination of processes including machine learning processes and human processes, wherein the human processes comprise computer code configured to perform operations and support interactions with a human. 
     
     
         16 . The computer program product of  claim 14 , wherein the set of computer-executable instructions further comprises instructions for:
 specifying a combination of stages of a workflow, the stages of the workflow including a generator stage and an assessor stage;   configuring inputs and outputs of the stages of the workflow; and   configuring connections between the stages of the workflow.   
     
     
         17 . The computer program product of  claim 16 , wherein the specified stages of a workflow include a combination of machine learning processes and human processes. 
     
     
         18 . The computer program product of  claim 1 , wherein the set of computer-executable instructions further comprises instructions for augmenting the set of output data based on the assessment to generate a set of synthetic data. 
     
     
         19 . The computer program product of  claim 14 , wherein the set of computer-executable instructions further comprises instructions for:
 packaging, by a dispatcher service, the generated set of output data as a task for presentation to a human specialist using a task user interface template; and   receiving a task result, by the dispatcher service, from the human specialist and returning the task result to the human process.   
     
     
         20 . The computer program product of  claim 19 , wherein the set of computer-executable instructions further comprises validating, by the dispatcher service, the task result.