Reducing processing for comparing large metadata sets

Abstract

A computer system compares databases, where each data record includes values for a plurality of fields, by applying database operations on database tables of the data records to reduce computations being performed. For each data record in a reference set, candidate matches are identified from among the data records in a target set by performing a join operation on the sets on one or more selected, normalized fields from the sets. This significantly reduced set of candidate matches can be further analyzed to determine how similar they are to the data record from the reference set.

What is claimed is: 
     
         1 . A computer system comprising:
 computer storage, wherein a first database and a second database are stored in the computer storage,   the first database comprising a first plurality of first data records, each first data record comprising a first plurality of first data fields storing metadata describing an item from among a first plurality of items,   the first plurality of data fields comprising at least a first identifier field, a first index field, and at least two more first metadata fields, each first identifier field for each first data record for an item storing a value representing an identifier for the item in the first database, each first index field for each first data record for an item storing a value representing a description of the item,   the second database comprising a second plurality of second data records, each second data record comprising a second plurality of second data fields storing metadata describing an item from among a second plurality of items,   the second plurality of data fields comprising at least a second identifier field, a second index field, and at least two more second metadata fields, each second identifier field for each second data record for an item storing a value representing an identifier for the item in the second database, each second index field for each second data record for an item storing a value representing a description of the item,   wherein, for similar items in the first database and the second database, the value representing the identifier for the similar item in the first identifier field may be different from the value representing the identifier for the similar item in the second identifier field,   a processing system comprising a memory and a processor, the processing system operable to access the computer storage and being configured by computer program instructions to be operative to:   normalize the values in the first index field of the first data records and the values in the second index field of the second data records;   select a candidate set based on the first database and the second database using the normalized first index field and the normalized second index field, to generate a table combining the first metadata fields and the second metadata fields, wherein the table comprises, for each data record in the first database having a similar data record in the second database, and for each similar data record, a row combining data fields for the data record from the first database with data fields for the similar data record from the second database, wherein the similar data record is a data record that has a value in the normalized second index field which is equal to the value in the normalized first index field for the data record in the first database;   for each row in the generated table, compare values of first metadata fields in the row to values of corresponding second metadata fields in the row to generate scores, and storing the scores as data fields in the row;   for each row in the generated table, determine whether the data record from the second database corresponding to the row is equivalent to the data record from the first database corresponding to the row, based on the generated scores stored in the row.   
     
     
         2 . The computer system of  claim 1 , wherein selecting the candidate set comprises performing a join operation of the first database and the second database on the normalized first index field and the normalized second index field to generate the table. 
     
     
         3 . The computer system of  claim 1 , wherein selecting the candidate set uses a plurality of normalized first index fields and a plurality of normalized second index fields. 
     
     
         4 . The computer system of  claim 3 , wherein selecting the candidate set comprises performing a first join operation of the first database and the second database on the normalized first index field and the normalized second index field to generate a first intermediate candidate table, performing a second join operation of the first database and the second database on a second normalized first index field and a second normalized second index field to generating a second intermediate candidate table, and performing a union of the first intermediate candidate table and the second intermediate candidate table to generate the table. 
     
     
         5 . The computer system of  claim 1 , wherein determining comprises applying first criteria to the scores indicative of a match of a first priority, and applying second criteria to the scores indicative of a match of a second priority lower than the first priority, and storing and indication of the match and the priority of the match. 
     
     
         6 . The computer system of  claim 2 , wherein comparing values of a first metadata field to a corresponding second metadata field in a row to generate a scores comprises applying an edit distance to text values stored in the first and second metadata fields. 
     
     
         7 . The computer system of  claim 6 , wherein comparing values of a first metadata field to a corresponding second metadata field in a row to generate a scores comprises determining if a text value stored in the first metadata field is contained in a text value stored in the second metadata field. 
     
     
         8 . An article of manufacture comprising:
 a computer storage device,   computer program instructions stored on the computer storage which, when processed by a computer, configures the computer to be comprising:   
       computer storage, wherein a first database and a second database are stored in the computer storage,
 the first database comprising a first plurality of first data records, each first data record comprising a first plurality of first data fields storing metadata describing an item from among a first plurality of items, 
 the first plurality of data fields comprising at least a first identifier field, a first index field, and at least two more first metadata fields, each first identifier field for each first data record for an item storing a value representing an identifier for the item in the first database, each first index field for each first data record for an item storing a value representing a description of the item, 
 the second database comprising a second plurality of second data records, each second data record comprising a second plurality of second data fields storing metadata describing an item from among a second plurality of items, 
 the second plurality of data fields comprising at least a second identifier field, a second index field, and at least two more second metadata fields, each second identifier field for each second data record for an item storing a value representing an identifier for the item in the second database, each second index field for each second data record for an item storing a value representing a description of the item, 
 wherein, for similar items in the first database and the second database, the value representing the identifier for the similar item in the first identifier field may be different from the value representing the identifier for the similar item in the second identifier field, 
 a processing system comprising a memory and a processor, the processing system operable to access the computer storage and being configured by computer program instructions to be operative to: 
 normalize the values in the first index field of the first data records and the values in the second index field of the second data records; 
 select a candidate set based on the first database and the second database using the normalized first index field and the normalized second index field, to generate a table combining the first metadata fields and the second metadata fields, wherein the table comprises, for each data record in the first database having a similar data record in the second database, and for each similar data record, a row combining data fields for the data record from the first database with data fields for the similar data record from the second database, wherein the similar data record is a data record that has a value in the normalized second index field which is equal to the value in the normalized first index field for the data record in the first database; 
 for each row in the generated table, compare values of first metadata fields in the row to values of corresponding second metadata fields in the row to generate scores, and storing the scores as data fields in the row; 
 for each row in the generated table, determine whether the data record from the second database corresponding to the row is equivalent to the data record from the first database corresponding to the row, based on the generated scores stored in the row. 
 
     
     
         9 . The article of manufacture of  claim 8 , wherein selecting the candidate set comprises performing a join operation of the first database and the second database on the normalized first index field and the normalized second index field to generate the table. 
     
     
         10 . The article of manufacture of  claim 8 , wherein selecting the candidate set uses a plurality of normalized first index fields and a plurality of normalized second index fields. 
     
     
         11 . The article of manufacture of  claim 10 , wherein selecting the candidate set comprises performing a first join operation of the first database and the second database on the normalized first index field and the normalized second index field to generate a first intermediate candidate table, performing a second join operation of the first database and the second database on a second normalized first index field and a second normalized second index field to generating a second intermediate candidate table, and performing a union of the first intermediate candidate table and the second intermediate candidate table to generate the table. 
     
     
         12 . The article of manufacture of  claim 8 , wherein determining comprises applying first criteria to the scores indicative of a match of a first priority, and applying a second criteria to the scores indicative of a match of a second priority lower than the first priority, and storing and indication of the match and the priority of the match. 
     
     
         13 . The article of manufacture of  claim 9 , wherein comparing values of a first metadata field to a corresponding second metadata field in a row to generate a scores comprises applying an edit distance to text values stored in the first and second metadata fields. 
     
     
         14 . The article of manufacture of  claim 9 , wherein comparing values of a first metadata field to a corresponding second metadata field in a row to generate a scores comprises determining if a text value stored in the first metadata field is contained in a text value stored in the second metadata field. 
     
     
         15 . A computer-implemented process performed by a computer program executing on a computer, the computer including computer storage, wherein a first database and a second database are stored in the computer storage,
 the first database comprising a first plurality of first data records, each first data record comprising a first plurality of first data fields storing metadata describing an item from among a first plurality of items,   the first plurality of data fields comprising at least a first identifier field, a first index field, and at least two more first metadata fields, each first identifier field for each first data record for an item storing a value representing an identifier for the item in the first database, each first index field for each first data record for an item storing a value representing a description of the item,   the second database comprising a second plurality of second data records, each second data record comprising a second plurality of second data fields storing metadata describing an item from among a second plurality of items,   the second plurality of data fields comprising at least a second identifier field, a second index field, and at least two more second metadata fields, each second identifier field for each second data record for an item storing a value representing an identifier for the item in the second database, each second index field for each second data record for an item storing a value representing a description of the item,   wherein, for similar items in the first database and the second database, the value representing the identifier for the similar item in the first identifier field may be different from the value representing the identifier for the similar item in the second identifier field,   the computer further including a processing system comprising a memory and a processor, the processing system operable to access the computer storage and being configured by computer program instructions to perform the process comprising:   normalizing the values in the first index field of the first data records and the values in the second index field of the second data records;   selecting a candidate set based on the first database and the second database using the normalized first index field and the normalized second index field, to generate a table combining the first metadata fields and the second metadata fields, wherein the table comprises, for each data record in the first database having a similar data record in the second database, and for each similar data record, a row combining data fields for the data record from the first database with data fields for the similar data record from the second database, wherein the similar data record is a data record that has a value in the normalized second index field which is equal to the value in the normalized first index field for the data record in the first database;   for each row in the generated table, comparing values of first metadata fields in the row to values of corresponding second metadata fields in the row to generate scores, and storing the scores as data fields in the row;   for each row in the generated table, determining whether the data record from the second database corresponding to the row is equivalent to the data record from the first database corresponding to the row, based on the generated scores stored in the row.   
     
     
         16 . The computer-implemented process of  claim 15 , wherein selecting the candidate set comprises performing a join operation of the first database and the second database on the normalized first index field and the normalized second index field to generate the table. 
     
     
         17 . The computer-implemented process of  claim 15 , wherein selecting the candidate set uses a plurality of normalized first index fields and a plurality of normalized second index fields. 
     
     
         18 . The computer-implemented process of  claim 17 , wherein selecting the candidate set comprises performing a first join operation of the first database and the second database on the normalized first index field and the normalized second index field to generate a first intermediate candidate table, performing a second join operation of the first database and the second database on a second normalized first index field and a second normalized second index field to generating a second intermediate candidate table, and performing a union of the first intermediate candidate table and the second intermediate candidate table to generate the table. 
     
     
         19 . The computer-implemented process of  claim 15 , wherein determining comprises applying first criteria to the scores indicative of a match of a first priority, and applying second criteria to the scores indicative of a match of a second priority lower than the first priority, and storing and indication of the match and the priority of the match. 
     
     
         20 . The computer-implemented process of  claim 16 , wherein comparing values of a first metadata field to a corresponding second metadata field in a row to generate a scores comprises applying an edit distance to text values stored in the first and second metadata fields.