High-accuracy data processing and machine learning techniques for sensitive data

Abstract

Flexible, high-accuracy data processing techniques and accompanying systems avoid criticality in intermediate computations through intelligent, low cost sanitization of data operations. A data processing operation including one or more plans is received, with each plan having a data operation described as a tree based-structure. The plans that are determined to create criticality on execution are sanitized by transforming the plan itself, ancestors, and/or children of the plan. Determining whether execution of a plan creates criticality is based on the determination of whether a set of criticality conditions includes data signals that are associated with the plan. After sanitization, the data processing operation can be fully executed without criticality arising in intermediate operations.

1 - 24 . (canceled) 
     
     
         25 . A computer-implemented method comprising:
 receiving a data processing operation defined by a first tree-based structure comprising a plurality of plans, the plans comprising a root plan, at least one inner plan, and a plurality of leaf plans, wherein the data processing operation when ordinarily executed creates criticality, and wherein each of the leaf plans is initially in a sanitized state such that execution of each such leaf plan on an individual basis does not create criticality;   determining that execution of a first one of the plans creates criticality, wherein the first plan is defined by a second tree-based structure comprising a plurality of sub-operations on data from a plurality of different execution environments;   sanitizing the first plan, taking into account the data from the different execution environments, by transforming the first plan, an ancestor plan of the first plan in the first tree-based structure, and/or a child plan of the first plan in the first tree-based structure such that execution of the first plan does not create criticality; and   following the sanitizing, executing the data processing operation, wherein execution of the data processing operation following the sanitizing no longer creates criticality.   
     
     
         26 . The method of  claim 25 , wherein determining that execution of the first plan creates criticality comprises identifying one or more data signals associated with the first plan and determining whether a set of criticality conditions includes one or more of the data signals. 
     
     
         27 . The method of  claim 26 , wherein sanitizing the first plan comprises:
 (a) determining a cost to remove from the first plan each data signal associated with the first plan that is included in the set of criticality conditions; and   (b) identifying a permutation of the data signals from step (a) that, when removed from the first plan, sanitize the first plan at a lowest cost compared to other permutations of the data signals.   
     
     
         28 . The method of  claim 25 , wherein sanitizing the first plan comprises applying to the first plan at least one transform operation in a set of transform operations, the set of transform operations being associated with (1) a data field in the first plan to be transformed and (2) a data signal to be removed from the first plan. 
     
     
         29 . The method of  claim 28 , wherein the set of transform operations comprises (1) a self transform in which a particular plan is transformed without transforming children of the particular plan, (2) an up transform in which each ancestor of the particular plan is transformed without transforming children of the particular plan, and/or (3) a root transform in which one or more transformations are reverted. 
     
     
         30 . The method of  claim 29 , wherein sanitizing the first plan further comprises performing a self transform on a name of the data field, the self transform comprising renaming the name of the data field in the first plan to create a renamed data field in the first plan. 
     
     
         31 . The method of  claim 30 , wherein sanitizing the first plan further comprises performing an up transform on a name of the data field, the up transform comprising changing, in each ancestor of the first plan, each reference to the name of the data field to a reference to the renamed data field. 
     
     
         32 . The method of  claim 31 , wherein sanitizing the first plan further comprises performing a root transform on a name of the data field, the root transform comprising reverting renaming and reference changing operations performed by the self transform and up transform of the first plan. 
     
     
         33 . The method of  claim 29 , wherein sanitizing the first plan further comprises performing a self transform on a value of the data field, the self transform comprising applying a lossless or lossy projection to the value of the data field in the first plan. 
     
     
         34 . The method of  claim 33 , wherein sanitizing the first plan further comprises performing an up transform on a value of the data field, the up transform comprising identifying operations in ancestors of the first plan that are potentially negatively affected by the self transform on the value of the data field. 
     
     
         35 . The method of  claim 34 , wherein sanitizing the first plan further comprises performing a root transform on a value of the data field, the root transform comprising either (1) reverting a lossless projection applied by the self transform of the first plan, (2) no operation, or (3) producing a set of values associated with a lossy projection applied by the self transform of the first plan. 
     
     
         36 . The method of  claim 25 , wherein execution of the first plan comprises training a data model, wherein the first plan is sanitized by applying a first transformation to inputs that are used for training the data model, and wherein execution of a second one of the plans comprises:
 using the data model to provide a prediction; and   applying the first transformation to inputs that are used by the data model to provide the prediction.   
     
     
         37 . The method of  claim 25 , further comprising determining that other plans in the plurality of plans create criticality on execution, and sanitizing the other plans prior to executing the data processing operation. 
     
     
         38 . A system comprising:
 a processor; and   a memory storing computer-executable instructions that, when executed by the processor, program the processor to perform the operations of:
 receiving a data processing operation defined by a first tree-based structure comprising a plurality of plans, the plans comprising a root plan, at least one inner plan, and a plurality of leaf plans, wherein the data processing operation when ordinarily executed creates criticality, and wherein each of the leaf plans is initially in a sanitized state such that execution of each such leaf plan on an individual basis does not create criticality; 
 determining that execution of a first one of the plans creates criticality, wherein the first plan is defined by a second tree-based structure comprising a plurality of sub-operations on data from a plurality of different execution environments; 
 sanitizing the first plan, taking into account the data from the different execution environments, by transforming the first plan, an ancestor plan of the first plan in the first tree-based structure, and/or a child plan of the first plan in the first tree-based structure such that execution of the first plan does not create criticality; and 
 following the sanitizing, executing the data processing operation, wherein execution of the data processing operation following the sanitizing no longer creates criticality. 
   
     
     
         39 . The system of  claim 38 , wherein determining that execution of the first plan creates criticality comprises identifying one or more data signals associated with the first plan and determining whether a set of criticality conditions includes one or more of the data signals. 
     
     
         40 . The system of  claim 39 , wherein sanitizing the first plan comprises:
 (c) determining a cost to remove from the first plan each data signal associated with the first plan that is included in the set of criticality conditions; and   (d) identifying a permutation of the data signals from step (a) that, when removed from the first plan, sanitize the first plan at a lowest cost compared to other permutations of the data signals.   
     
     
         41 . The system of  claim 38 , wherein sanitizing the first plan comprises applying to the first plan at least one transform operation in a set of transform operations, the set of transform operations being associated with (1) a data field in the first plan to be transformed and (2) a data signal to be removed from the first plan. 
     
     
         42 . The system of  claim 41 , wherein the set of transform operations comprises (1) a self transform in which a particular plan is transformed without transforming children of the particular plan, (2) an up transform in which each ancestor of the particular plan is transformed without transforming children of the particular plan, and/or (3) a root transform in which one or more transformations are reverted. 
     
     
         43 . The system of  claim 38 , wherein execution of the first plan comprises training a data model, wherein the first plan is sanitized by applying a first transformation to inputs that are used for training the data model, and wherein execution of a second one of the plans comprises:
 using the data model to provide a prediction; and   applying the first transformation to inputs that are used by the data model to provide the prediction.   
     
     
         44 . The system of  claim 38 , wherein the operations further comprise determining that other plans in the plurality of plans create criticality on execution, and sanitizing the other plans prior to executing the data processing operation.