Automatic entity resolution with rules detection and generation system

Abstract

Entity resolution (i.e., record linkage) involves the analysis/discovering of datasets that refer to the same real world entity. Analysis typically involves transformation and comparison of different fields of the dataset followed by the application of often domain/data specific logic for determining datasets that refer to the same real world entity (e.g., person). Consider, a bulk mailing of product catalogs to potential customers. Some individuals may have numerous public records that identify the individual differently. Illustratively, several records associated with Jane Doe at her current home address may exist: one record with her name listed as J. Doe, a second record as Jane H. Doe, a third record as Doe, Jane, and a fourth record as Jan Doe (a misspelling). Conceivably, the bulk mailing could unwittingly send multiple catalogs to Jane Doe at her current address, one for each name variation. The entity resolution process described herein can overcome such problems.

1 . A method of training a system to detect whether records belong to a same entity or a different entity, the method comprising:
 receiving, as input, a plurality of records known to be associated with a same entity, each record including a plurality of fields containing data about the entity;   defining at least one link feature by pairing a field in one of the records with a field in another of the records;   associating with the paired fields a similarity metric used to determine a degree of similarity between the data in those paired fields;   applying the similarity metric of the at least one link feature to the data in the paired fields to produce a link feature value for each link feature;   generating a linkage fata instance comprised of multiple defined link features that are used to determine whether records of unknown association belong to a same individual;   applying the link features to two records of unknown association to produce a linkage data instance;   using the linkage data instance to determine whether the two records are associated with a same individual; and   outputting an indication as to whether the two records of unknown association are associated with the same individual.   
     
     
         2 - 27 . (canceled)