Data curation system with version control for workflow states and provenance

Abstract

A data curation system that includes various methods to enable efficient reuse of human and machine effort. To reuse effort, various facilities are presented that model, save, and allow the querying of provenance and state information of a curation workflow and allow for incremental, stateful transitions of the data and the metadata.

What is claimed is: 
     
         1 . A method of provenance creation, tracking, and re-use as part of improved curation of large scale data sets, comprising:
 operating software on a computer system for data curation, the software performing data curation actions of data loading, transformation, and linkage;   wherein data loading comprises:
 identifying a new data source external to the operating computer system, wherein the data source comprises 8,000 or more records, and each record is a key-value pair, wherein every key corresponds to an attribute or column of the new data source; and 
   loading the records of the new data source into storage accessed by the operating computer system, wherein storage is within a large-scale relational database management system or Hadoop cluster;   wherein linkage comprises:
 posing linkage questions to data experts; 
 obtaining linkage opinions from data experts; 
 generating linkage suggestions based on machine learning of a linkage model; 
 establishing linkages facts by explicit approval of a linkage suggestion by a system operator, or bulk approval based on system operator configured criteria balancing linkage model confidence and data expert opinion consensus; 
 wherein linkage further comprises attribute linkage for schema mapping between different data sources, and record linkage for deduplication; 
 and wherein a linkage fact identifies two attributes or two different records which are linked as equivalent; 
   wherein transformation comprises:
 applying a transformation script language or extract, transform, load (ETL) tools to create new attributes derived from existing attributes, or modify existing attributes and records; 
   wherein each data curation action comprises the following steps:
 initiation by the system operator or scheduled task configured by the system operator; 
 action processing to produce a Curation Proposal comprising a set of suggested changes and confidence for each suggestion; 
 presenting the curation proposal to the system operator, wherein the system operator may approve or reject individual suggestions within the curation proposal or apply a selection criteria to approve or reject suggestions in bulk, forming candidate changes as the set of accepted suggestions from the curation proposal; 
 computing an updated proposal by propagating dependencies from the candidate changes; 
 approving or rejecting changes within the updated proposal based on a dependency processing mode (DPM) selected by the system operator, and creating final changes as the changes approved based on the DPM; and 
 applying the final changes to a current curation state to create a new curation state. 
   wherein each curation state includes:
 one or more data sources; 
 one or more linkage facts about attributes and records of the data sources; 
 a linkage system state comprising training data, linkage models, and any intermediate states computed during generation of linkage suggestions; 
 one or more linkage questions; 
 one or more linkage opinions; 
   wherein each curation state may be stored independent of a previous curation state or as a set of changes from the previous curation state;   for every curation state change, recording provenance metadata about the change, wherein provenance metadata comprises:
 which curation data action occurred to cause the change; 
 which system operator took the curation data action action causing the change; 
 what conditions are required for the change to remain valid, wherein linkage actions remain valid based on:
 explicit approval actions by a system operator remain valid until explicit approval is removed; 
 bulk approval actions remain valid as long as criteria for bulk approval remains met; 
 linkage models based on training data remain valid while a configured critical mass of training facts remain valid; and 
 transformation actions remain valid as long as inputs to the transformation action remain valid; 
 
   forming a directed acyclic graph (DAG) based on recorded provenance, where each node of the DAG without any in-edges represents a curation action, and each node with in-edges represents an individual curation state element;   traversing the DAG to determine which curation state elements depend from an individual element;   using the DAG to determine invalidated dependencies by calculating the set of descendants of all proposed changes when propagating changes to a curation state;   wherein the DPM applied to approve or reject changes in the updated proposal is selected from restrict, propagate, or override, and wherein restrict rejects any changes that change any existing facts, propagate uses provenance information of any elements being changed to compute further changes whose application makes the provenance of all elements consistent, and override updates any elements rendered invalid by the candidate changes to being approved by the system operator:   selecting a prior curation state to create a branch, updating the current curation state to the prior curation state, and tracking all curation state changes in the branch as children from the prior curation state; and   merging two different branches by:
 identifying which branch is a deeper descendant from a least common ancestor curation state of both branches, and using the identified branch as a base; and 
 applying, in sequential order from the least common ancestor curation state, all curation state changes in the non-identified branch to the identified branch using a propagate DPM.