Mapping instances of a dataset within a data management system

Abstract

Mapping data stored in a data storage system for use by a computer system includes processing specifications of dataflow graphs that include nodes representing computations interconnected by links representing flows of data. At least one of the dataflow graphs receives a flow of data from at least one input dataset and at least one of the dataflow graphs provides a flow of data to at least one output dataset. A mapper identifies one or more sets of datasets. Each dataset in a given set matches one or more criteria for identifying different versions of a single dataset. A user interface is provided to receive a mapping between at least two datasets in a given set. The mapping received over the user interface is stored in association with a dataflow graph that provides data to or receives data from the datasets of the mapping.

1 . A method for mapping data stored in a data storage system for use by a computer system, the method including:
 processing specifications of dataflow graphs that include nodes representing computations interconnected by links representing flows of data, with at least one of the dataflow graphs receiving a flow of data from at least one input dataset and at least one of the dataflow graphs providing a flow of data to at least one output dataset;   identifying one or more sets of datasets, where each dataset in a given set matches one or more criteria for identifying different versions of a single dataset, each version of the single dataset representing data received or provided by a different one of the dataflow graphs;   providing a user interface to receive a mapping between at least two datasets in a given set; and   storing the mapping received over the user interface in association with a dataflow graph that provides data to or receives data from the datasets of the mapping.   
   
   
       2 . The method of  claim 1 , including presenting the set over the user interface. 
   
   
       3 . The method of  claim 1 , including presenting over the user interface a list of possible mappings ordered according to a quantification of a match to the one or more criteria. 
   
   
       4 . The method of  claim 3 , wherein the list of possible mappings includes candidates that are more likely to be an instance of a given dataset ordered higher in the list. 
   
   
       5 . The method of  claim 3 , wherein one of the criteria is built into a mapper that identifies the one or more sets of datasets. 
   
   
       6 . The method of  claim 3 , wherein one of the criteria is received from the user interface. 
   
   
       7 . The method of  claim 3 , wherein at least one of the possible mappings indicates a component of a dataflow graph that represents a dataset, and at least one of the possible mappings indicates a component of a dataflow graph that does not represent a dataset. 
   
   
       8 . The method of  claim 1 , wherein a sub-graph of a dataflow graph including multiple components represents a dataset. 
   
   
       9 . The method of  claim 8 , wherein the sub-graph includes a data component. 
   
   
       10 . The method of  claim 8 , wherein the sub-graph includes an executable component. 
   
   
       11 . The method of  claim 1 , wherein identifying one or more sets of datasets includes using heuristics for determining if a dataset in a given set has one or more characteristics in common with another dataset. 
   
   
       12 . The method of  claim 11 , wherein the characteristics include the quantity of bytes and records in a representation of the dataset. 
   
   
       13 . The method of  claim 11 , wherein the characteristics include the name of a representation of the dataset. 
   
   
       14 . The method of  claim 11 , wherein the characteristics include the date of creation of a representation of the dataset. 
   
   
       15 . The method of  claim 11 , wherein the characteristics include the data format of a representation of the dataset. 
   
   
       16 . The method of  claim 1 , wherein at least one of the datasets of the mapping belongs to a group of datasets known to a data management system. 
   
   
       17 . The method of  claim 1 , further including providing a format mapping between datasets in a given set. 
   
   
       18 . The method of  claim 1 , wherein the mapping includes an identifier that points to a record in the data management system that keeps track of the dataset. 
   
   
       19 . The method of  claim 1 , further including updating the mapping based on a change in a dataset. 
   
   
       20 . A system for mapping data stored in a data storage system, the system including
 a data storage system storing specifications of dataflow graphs that include nodes representing computations interconnected by links representing flows of data, with at least one of the dataflow graphs receiving a flow of data from at least one input dataset and at least one of the dataflow graphs providing a flow of data to at least one output dataset;   a mapper that identifies one or more sets of datasets associated with the dataflow graphs, where each dataset in a given set matches one or more criteria for identifying different versions of a single dataset, each version of the single dataset representing data received or provided by a different one of the dataflow graphs;   a user interface that receives a mapping between at least two datasets in a given set, and stores the mapping in the data storage system in association with a dataflow graph that provides data to or receives data from the datasets of the mapping.   
   
   
       21 . The system of  claim 20 , wherein the user interface presents the set. 
   
   
       22 . The system of  claim 20 , wherein the user interface presents a list of possible mappings ordered according to a quantification of a match to the one or more criteria. 
   
   
       23 . The system of  claim 22 , wherein the list of possible mappings includes candidates that are more likely to be an instance of a given dataset ordered higher in the list. 
   
   
       24 . The system of  claim 22 , wherein one of the criteria is built into the mapper. 
   
   
       25 . The system of  claim 22 , wherein one of the criteria is received by the user interface. 
   
   
       26 . The system of  claim 22 , wherein at least one of the possible mappings indicates a component of a dataflow graph that represents a dataset, and at least one of the possible mappings indicates a component of a dataflow graph that does not represent a dataset. 
   
   
       27 . The system of  claim 20 , wherein a sub-graph of a dataflow graph including multiple components represents a dataset. 
   
   
       28 . The system of  claim 27 , wherein the sub-graph includes a data component. 
   
   
       29 . The system of  claim 27 , wherein the sub-graph includes an executable component. 
   
   
       30 . The system of  claim 20 , wherein the mapper uses heuristics for determining if a dataset in a given set has one or more characteristics in common with another dataset. 
   
   
       31 . The system of  claim 30 , wherein the characteristics include the quantity of bytes and records in a representation of the dataset. 
   
   
       32 . The system of  claim 30 , wherein the characteristics include the name of a representation of the dataset. 
   
   
       33 . The system of  claim 30 , wherein the characteristics include the date of creation of a representation of the dataset. 
   
   
       34 . The system of  claim 30 , wherein the characteristics include the data format of a representation of the dataset. 
   
   
       35 . The system of  claim 20 , wherein at least one of the datasets of the mapping belongs to a group of datasets known to a data management system. 
   
   
       36 . The system of  claim 20 , wherein the mapper generates a format mapping between datasets in a given set. 
   
   
       37 . The system of  claim 20 , wherein the mapping includes an identifier that points to a record in the data management system that keeps track of the dataset. 
   
   
       38 . The system of  claim 20 , wherein the mapper updates the mapping based on a change in a dataset. 
   
   
       39 . A system for mapping data stored in a data storage system, the system including:
 means for processing specifications of dataflow graphs that include nodes representing computations interconnected by links representing flows of data, with at least one of the dataflow graphs receiving a flow of data from at least one input dataset and at least one of the dataflow graphs providing a flow of data to at least one output dataset;   means for identifying one or more sets of datasets, where each dataset in a given set matches one or more criteria for identifying different versions of a single dataset each version of the single dataset representing data received or provided by a different one of the dataflow graphs;   means for providing a user interface to receive a mapping between at least two datasets in a given set; and   means for storing the mapping received over the user interface in association with a dataflow graph that provides data to or receives data from the datasets of the mapping.   
   
   
       40 . A computer-readable medium storing a computer program for mapping data stored in a data storage system, the computer program including instructions for causing a computer to:
 process specifications of dataflow graphs that include nodes representing computations interconnected by links representing flows of data, with at least one of the dataflow graphs receiving a flow of data from at least one input dataset and at least one of the dataflow graphs providing a flow of data to at least one output dataset;   identify one or more sets of datasets, where each dataset in a given set matches one or more criteria for identifying different versions of a single dataset, each version of the single dataset representing data received or provided by a different one of the dataflow graphs;   provide a user interface to receive a mapping between at least two datasets in a given set; and   store the mapping received over the user interface in association with a dataflow graph that provides data to or receives data from the datasets of the mapping.   
   
   
       41 . The method of  claim 1 , wherein each version of a single dataset is associated with a different graph, graph subset, or executable component. 
   
   
       42 . The method of  claim 1 , wherein each version of a single dataset is stored in a different location associated with the data storage system. 
   
   
       43 . The method of  claim 1 , wherein each version of a single dataset is interpreted using a different data storage format. 
   
   
       44 . The method of  claim 1 , wherein each version of a single dataset is accessed using a parameter that varies between executions of the dataflow graph.