Cleansing a Database System to Improve Data Quality

Abstract

According to one embodiment of the present invention, a system controls cleansing of data within a database system, and comprises a computer system including at least one processor. The system receives a data set from the database system, and one or more features of the data set are selected for determining values for one or more characteristics of the selected features. The determined values are applied to a data quality estimation model to determine data quality estimates for the data set. Problematic data within the data set are identified based on the data quality estimates, where the cleansing is adjusted to accommodate the identified problematic data. Embodiments of the present invention further include a method and computer program product for controlling cleansing of data within a database system in substantially the same manner described above.

1 - 8 . (canceled) 
     
     
         9 . A system for controlling cleansing of data within a database system comprising:
 a computer system including at least one processor configured to:
 receive a data set from the database system; 
 select one or more features of the data set and determine values for one or more characteristics of the selected features; 
 apply the determined values to a data quality estimation model to determine data quality estimates for the data set; 
 identify problematic data within the data set based on the data quality estimates; and 
 adjust the cleansing to accommodate the identified problematic data. 
   
     
     
         10 . The system of  claim 9 , wherein the at least one processor is further configured to:
 partition the data set into a plurality of groups with each group including data of a similar data quality.   
     
     
         11 . The system of  claim 10 , wherein the applying the determined values further includes:
 applying the determined values associated with each group to the data quality estimation model to determine data quality estimates for each group.   
     
     
         12 . The system of  claim 11 , wherein the identifying problematic data further includes:
 identifying one or more groups with problematic data based on the data quality estimates for each group, wherein the identified groups include data quality estimates with the lowest values.   
     
     
         13 . The system of  claim 9 , wherein the cleansing is performed in accordance with one or more rules, and the adjusting the cleansing further includes at least one of:
 modifying the one or more rules for the cleansing to accommodate the identified problematic data; and   creating one or more new rules for the cleansing to accommodate the identified problematic data.   
     
     
         14 . The system of  claim 9 , wherein the data quality estimation model includes a regression model. 
     
     
         15 . The system of  claim 9 , wherein the cleansing includes matching and linking phases to identify duplicate data, and the at least one processor is further configured to:
 remove the identified problematic data from the matching and linking phases to enhance the identification of duplicate data.   
     
     
         16 . The system of  claim 9 , wherein the applying the determined values further includes:
 applying the determined values associated with corresponding data fields of the data set to the data quality estimation model to determine data quality estimates for each field;   wherein the identifying problematic data further includes:   identifying one or more data fields with problematic data based on the data quality estimates for each data field, wherein the identified data fields include data quality estimates with the lowest values.   
     
     
         17 . A computer program product for controlling cleansing of data within a database system comprising:
 a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising computer readable program code configured to:
 receive a data set from the database system; 
 select one or more features of the data set and determine values for one or more characteristics of the selected features; 
 apply the determined values to a data quality estimation model to determine data quality estimates for the data set; 
 identify problematic data within the data set based on the data quality estimates; and 
 adjust the cleansing to accommodate the identified problematic data. 
   
     
     
         18 . The computer program product of  claim 17 , wherein the computer readable program code further includes computer readable program code configured to:
 partition the data set into a plurality of groups with each group including data of a similar data quality.   
     
     
         19 . The computer program product of  claim 18 , wherein the applying the determined values further includes:
 applying the determined values associated with each group to the data quality estimation model to determine data quality estimates for each group.   
     
     
         20 . The computer program product of  claim 19 , wherein the identifying problematic data further includes:
 identifying one or more groups with problematic data based on the data quality estimates for each group, wherein the identified groups include data quality estimates with the lowest values.   
     
     
         21 . The computer program product of  claim 17 , wherein the cleansing is performed in accordance with one or more rules, and the adjusting the cleansing further includes at least one of:
 modifying the one or more rules for the cleansing to accommodate the identified problematic data; and   creating one or more new rules for the cleansing to accommodate the identified problematic data.   
     
     
         22 . The computer program product of  claim 17 , wherein the data quality estimation model includes a regression model. 
     
     
         23 . The computer program product of  claim 17 , wherein the cleansing includes matching and linking phases to identify duplicate data, and the computer readable program code further includes computer readable program code configured to:
 remove the identified problematic data from the matching and linking phases to enhance the identification of duplicate data.   
     
     
         24 . The computer program product of  claim 17 , wherein the applying the determined values further includes:
 applying the determined values associated with corresponding data fields of the data set to the data quality estimation model to determine data quality estimates for each field;   wherein the identifying problematic data further includes:   identifying one or more data fields with problematic data based on the data quality estimates for each data field, wherein the identified data fields include data quality estimates with the lowest values.