Automated adaptive data analysis using dynamic data quality assessment

Abstract

In general, embodiments of the present invention provide systems, methods and computer readable media for automated dynamic data quality assessment. One aspect of the subject matter described in this specification includes the actions of receiving a data quality job including a new data sample; and, if the new data sample is determined to be added to a reservoir of data samples, sending a quality verification request to an oracle; receiving a new data sample quality estimate from the oracle; and adding the new data sample and estimate to the reservoir. A second aspect of the subject matter includes the actions of receiving, from a predictive model, a judgment associated with a new data sample; analyzing the new data sample based in part on the judgment to determine whether to send a new data sample quality verification request to an oracle; and, if a new data sample quality estimate is received from the oracle, determining whether to add the new data sample and the judgment to the reservoir.

1 - 24 . (canceled) 
     
     
         25 . An apparatus comprising at least one processor and at least one memory storing instructions that, with the at least one processor, cause the apparatus to:
 determine whether to add a new data sample and a judgment associated with the new data sample to a reservoir of data samples in response to receiving a data quality estimate for the new data sample, wherein the judgment is generated by a predictive model, wherein the determination is based on one of whether the new data sample statistically belongs in the data reservoir or whether the judgment is associated with a high confidence value; and   update reservoir summary statistics for the reservoir of data samples in response to determining whether to add the new data sample and the judgment associated with the new data sample to the reservoir of data samples.   
     
     
         26 . The apparatus of  claim 25 , wherein the judgment includes a confidence value. 
     
     
         27 . The apparatus of  claim 25 , wherein the new data sample is collected from a data stream. 
     
     
         28 . The apparatus of  claim 25 , wherein the judgment is generated based on a feature vector associated with the new data sample. 
     
     
         29 . The apparatus of  claim 28 , wherein the feature vector represents an optimal view of the new data sample. 
     
     
         30 . The apparatus of  claim 25 , wherein the new data sample comprises a particular data type. 
     
     
         31 . The apparatus of  claim 30 , wherein the reservoir of data samples is identified based at least in part on the particular data type. 
     
     
         32 . The apparatus of  claim 25 , wherein the data quality estimate is received from an oracle. 
     
     
         33 . A non-transitory computer-readable storage medium storing computer code that, when executed by at least one process of an apparatus, cause the apparatus to:
 determine whether to add a new data sample and a judgment associated with the new data sample to a reservoir of data samples in response to receiving a data quality estimate for the new data sample, wherein the judgment is generated by a predictive model, wherein the determination is based on one of whether the new data sample statistically belongs in the data reservoir or whether the judgment is associated with a high confidence value; and   update reservoir summary statistics for the reservoir of data samples in response to determining whether to add the new data sample and the judgment associated with the new data sample to the reservoir of data samples.   
     
     
         34 . The computer-readable storage medium of  claim 33 , wherein the judgment includes a confidence value. 
     
     
         35 . The computer-readable storage medium of  claim 33 , wherein the new data sample is collected from a data stream. 
     
     
         36 . The computer-readable storage medium of  claim 33 , wherein the judgment is generated based on a feature vector associated with the new data sample. 
     
     
         37 . The computer-readable storage medium of  claim 36 , wherein the feature vector represents an optimal view of the new data sample. 
     
     
         38 . The computer-readable storage medium of  claim 33 , wherein the new data sample comprises a particular data type. 
     
     
         39 . The computer-readable storage medium of  claim 38 , wherein the reservoir of data samples is identified based at least in part on the particular data type. 
     
     
         40 . The computer-readable storage medium of  claim 33 , wherein the data quality estimate is received from an oracle. 
     
     
         41 . A computer-implemented method, comprising:
 determining whether to add a new data sample and a judgment associated with the new data sample to a reservoir of data samples in response to receiving a data quality estimate for the new data sample, wherein the judgment is generated by a predictive model, wherein the determination is based on one of whether the new data sample statistically belongs in the data reservoir or whether the judgment is associated with a high confidence value; and   updating reservoir summary statistics for the reservoir of data samples in response to determining whether to add the new data sample and the judgment associated with the new data sample to the reservoir of data samples.   
     
     
         42 . The method of  claim 41 , wherein the judgment is generated based on a feature vector associated with the new data sample. 
     
     
         43 . The method of  claim 41 , wherein the new data sample comprises a particular data type. 
     
     
         44 . The method of  claim 43 , wherein the reservoir of data samples is identified based at least in part on the particular data type.