Data curation system with version control for workflow states and provenance

Abstract

A data curation system that includes various methods to enable efficient reuse of human and machine effort. To reuse effort, various facilities are presented that model, save, and allow the querying of provenance and state information of a curation workflow and allow for incremental, stateful transitions of the data and the metadata.

What is claimed is: 
     
         1 . A system for data curation with version control comprising:
 a system operator;   data experts;   a curation processes module;   a state creation and manipulation module;   an update handler module; and   a curation states and provenance datastore;   wherein said curation processes module is operable to
 send questions to said data experts and 
 receive opinions from said data experts about said data curation; 
   wherein said curation processes module is operable to
 output a curation proposal to said system operator and 
 output state changes and provenance to said state creation and manipulation module; 
   wherein said state creation and manipulation module is operable to
 output candidate changes to said update handler module; 
   wherein said update handler module is operable to
 output an update proposal to said system operator, 
 input update approvals from said system operator, 
 output final changes to said state creation and manipulation module; 
   wherein said state creation and manipulation module is operable to
 output new states and provenance to said curation states and provenance datastore, 
 wherein said new states and provenance is a child state of the former parent state, wherein said parent state, said child state, and information about the action that the changed said parent state to said child state are all stored in said curation states and provenance datastore, thereby enabling said version control for said data curation. 
   
     
     
         2 . The system of  claim 1  wherein said action is a curation data action initiated by said system operator via said curation process module, wherein said curation data action is operable to act on the underlying data that is being curated. 
     
     
         3 . The system of  claim 1  wherein said action is a state history action initiated by said system operator via said state creation and manipulation module, wherein said state history action is operable to manipulate the state history. 
     
     
         4 . The system of  claim 3  wherein said state history action is selected from the group consisting of move to other state, branch, merge, rebase. 
     
     
         5 . A method for data curation with version control comprising the steps of:
 defining a current curation state as a parent state of a data source to be curated, wherein said parent state includes the following elements:
 said data source; 
 facts about which objects in said data source are linked; 
 linkage system state; 
 questions posed to data experts about said data source; 
 answers from said data experts about said data source; 
   writing said current curation state to a datastore;   initiating a curation data action on said parent state;   creating a child state of said parent state based on said curation data action;   identifying which of said elements would be changed by said curation data action;   identifying all of the changes between said parent state and said child state;   identifying provenance information for said changes; and   writing said child state and said provenance information to said datastore.   
     
     
         6 . The method of  claim 5  wherein said curation data action is loading a new data source or adding data to an existing data source. 
     
     
         7 . The method of  claim 5  wherein said curation data action is transforming a data source, either in place or to create a new data source. 
     
     
         8 . The method of  claim 5  wherein said curation data action is adding object linkage facts. 
     
     
         9 . The method of  claim 5  wherein said curation data action is adding linkage questions. 
     
     
         10 . The method of  claim 5  wherein said curation data action is accepting linkage opinions. 
     
     
         11 . The method of  claim 5  wherein one or more of said curation data actions are grouped in a batch and said child state incorporates said batch of said curation data actions. 
     
     
         12 . The method of  claim 5  including the steps of:
 generating a curation proposal, said curation proposal received from a first computer module and sent
 to a system operator in human-readable form and 
 to a second computer module in machine-readable form; and 
 
 approving said curation proposal. 
 
     
     
         13 . The method of  claim 5  wherein said curation data action includes a dependency processing mode selected from the group consisting of restrict, propagate, or override. 
     
     
         14 . The method of  claim 5  wherein said curation data action includes a preview mode. 
     
     
         15 . The method of  claim 5  where said provenance information includes which of said curation actions produced said changes and which conditions have to hold in order for each of said elements to remain true. 
     
     
         16 . The method of  claim 5  repeated to form multiple states, such as a first action that changes a state one to a state two and a second action that changes said state two to a state three, said state two and said state three being child states of said state one and said state two, respectively; and including the steps of:
 forming a directed acyclic graph from said provenance information of each of said states; and 
 using graph traversal on said directed acyclic graph to determine
 which of said states each of said elements depends upon and 
 which of said elements each of said states depends upon. 
 
 
     
     
         17 . A method for curation state history manipulation with version control comprising the steps of:
 defining a current curation state as a parent state of a data source to be curated, wherein said parent state includes the following elements:
 said data source; 
 facts about which objects in said data source are linked; 
 linkage system state; 
 questions posed to data experts about said data source; 
 answers from said data experts about said data source; 
 writing said current curation state to a datastore; 
 initiating a state history action on said parent state; 
 creating a child state of said parent state based on said state history action; 
 identifying which of said elements would be changed by said state history action; 
 identifying all of the changes between said parent state and said child state; 
 identifying provenance information for said changes; and 
 writing said child state and said provenance information to said datastore. 
   
     
     
         18 . The method of  claim 17  where said state history action is move to other state. 
     
     
         19 . The method of  claim 17  where said state history action is branch. 
     
     
         20 . The method of  claim 17  where said state history action is merge. 
     
     
         21 . The method of  claim 17  where said state history action is rebase. 
     
     
         22 . A method of reusing human and machine effort in a data curation system comprising:
 creating a current curation state as a parent state of a data source to be curated, wherein said parent state includes the following elements:
 said data source; 
 facts about which objects in said data source are linked; 
 linkage system state; 
 questions posed to data experts about said data source; 
 answers from said data experts about said data source; 
   writing said current curation state to a datastore;   defining an action to take said parent state   creating a child state of said parent state based on said action;   identifying which of said elements would be changed by said action;   identifying all of the changes between said parent state and said child state;   identifying provenance information for said changes;   determining which of said states each of said elements depends upon and which of said elements each of said states depends upon, thereby allowing a system operator of said system to reuse previous work in implementing said action;   taking said action; and   writing said child state and said provenance information to said datastore.   
     
     
         23 . The method of  claim 22  wherein said action is loading a new data source or adding data to an existing data source. 
     
     
         24 . The method of  claim 22  wherein said action is transforming a data source, either in place or to create a new data source. 
     
     
         25 . The method of  claim 22  wherein said action is adding object linkage facts. 
     
     
         26 . The method of  claim 22  wherein said action is adding linkage questions. 
     
     
         27 . The method of  claim 22  wherein said action is accepting linkage opinions. 
     
     
         28 . The method of  claim 22  wherein said action is moving to another of said states. 
     
     
         29 . The method of  claim 22  wherein said action is branching one of said states 
     
     
         30 . The method of  claim 22  wherein said action is merging two or more of said states. 
     
     
         31 . The method of  claim 22  wherein said action is rebasing two or more of said states.