Data classification

Abstract

A method provides for classifying data fields of a dataset. A classifier configured for determining confidence values for a plurality of data classes for the data fields may be applied. Using the confidence values, data class candidates may be identified. Data fields may be determined for which a plurality of data class candidates is identifiable. Using previous user-selected data class assignments, a probability may be determined for the data class candidates that the respective data class candidate is a data class to which the respective data field is to be assigned. The data fields may be classified using the probabilities to select for the data fields a data class from the data class candidates. The dataset may be provided with metadata identifying for the data fields the data classes to which the respective data fields are assigned.

What is claimed is: 
     
       1. A computer-implemented method for automatically classifying data fields of a dataset, the method comprising:
 providing training datasets, each training dataset comprising one or more training first data fields and previous user-selected data class assignments for the one or more training first data fields, each training first data field being assigned with a plurality of data class candidates; and 
 executing a machine learning algorithm on the training datasets for generating a machine learning model, wherein executing the machine learning algorithm comprises:
 retrieving, from a computer readable storage medium, the previous user-selected data class assignments for the one or more training first data fields 
 calculating, using the machine learning model, clusters of data classes being assigned as data class candidates for the same training data fields of the training datasets; 
 calculating, using the machine learning model, probabilities for each of the data class candidates assigned to the training first data fields that a respective data class candidate is a data class to which the respective training first data field is assigned taking into the clusters of data classes assigned as data class candidates to adjacent training data fields within a predefined range of interest around the respective training first data field; and 
 comparing the calculated probabilities to the retrieved user-selected data class assignments; 
 
 storing, in the computer readable storage medium, confidence values for a plurality of data classes, wherein the confidence values are determined by applying a classifier to data fields of a dataset, wherein the classifier is configured for determining the confidence values independently from one another for a plurality of data fields, wherein the confidence values identify a level of confidence that a respective data field belongs to a respective data class, and wherein the data class for which the confidence value exceeds a predefined threshold is identified as a data class candidate for the respective data field for which the respective confidence value is determined; 
 calculating, by a processor, and storing, in the computer readable storage medium, first data fields for which the plurality of data class candidates are identifiable; 
 calculating, by the machine learning model trained using previous user-selected data class assignments, and storing, in the computer readable storage medium, a probability for the data class candidates identified for the first data fields that the respective data class candidate is a data class to which a respective first data field is to be assigned; 
 storing, in the computer readable storage medium, classifications for the stored first data fields, wherein the classifications are determined, by the processor, using the stored probabilities for the data class candidates to select for the stored first data fields a data class from the data class candidates for the respective first data field to which the respective first data field being assigned; and 
 storing, in the computer readable storage medium, the dataset with metadata identifying for the classified data fields of the dataset the data classes to which the respective classified data fields are assigned; 
 importing the classified dataset into a target database in the computer readable storage medium; 
 organizing the target database using the target data model defining a class-based arrangement of data fields; 
 rearranging data fields of the dataset according to the target data model using metadata identifying data classes to which the data fields of the dataset are assigned; 
 adding the rearranged dataset to the target database in accordance with the target data model; 
 executing a search query on the classified dataset with the metadata identifying data classes using a data class identifier as a search parameter of the search query; and 
 providing a search result, for the search query, comprise one or more data values comprised by datasets from data fields assigned to the data classes identified by the data class identifiers. 
 
     
     
       2. The method of  claim 1 , wherein the data fields of the dataset are organized in a form of a data table using a row-oriented data model comprising data fields in form of columns. 
     
     
       3. The method of  claim 1 , wherein the dataset is provided in a form of a structured document defining a set of entities with the data fields being provided in form of attributes assigned to the entities. 
     
     
       4. The method of  claim 1 , wherein the determining of the probabilities comprises:
 providing the first data fields and identifiers of the data class candidates of the plurality of data class candidates for each of the first data fields as input data to the machine learning model; and 
 in response to the providing of the input data, receiving from the machine learning model as output data, the probability for each of the data class candidates that the respective data class candidate is the data class to which the respective first data field is to be assigned. 
 
     
     
       5. The method of  claim 4 :
 wherein one or more of the training datasets comprises one or more training second data fields and previous user-selected data class assignments for the one or more training second data fields, each training second data field being assigned with a single data class candidate; and 
 the method further comprising:
 determining second data fields of the dataset for which the single data class candidate being identifiable; and 
 classifying the second data fields, the classifying of the second data fields comprising assigning each of the second data fields to the single data class identified as the data class candidate for the respective second data field. 
 
 
     
     
       6. The method of  claim 5 , wherein the range of interest extends symmetrically around the respective training first data field. 
     
     
       7. The method of  claim 5 , wherein the range of interest extends asymmetrically around the respective training first data field. 
     
     
       8. The method of  claim 4 , wherein the learning algorithm is configured for:
 determining for each training dataset a pattern of data classes assigned to the data fields comprised by a respective training dataset; and 
 determining, using a sequential pattern algorithm, sequential pattern rules indicating which data classes to be found together in a same dataset. 
 
     
     
       9. The method of  claim 8 , the sequential pattern algorithm is selected from a group of association rule algorithms consisting of: an apriori algorithm, an eclat algorithm, and an FP-growth algorithm. 
     
     
       10. The method of  claim 4 , wherein the generating of the machine learning model comprises providing one or more logical data models or industry models as a further input to the learning algorithm. 
     
     
       11. The method of  claim 1 , wherein the determining of the probabilities comprises analyzing similarities between metadata assigned to the first data fields and metadata assigned to the data class candidates. 
     
     
       12. The method of  claim 1 , wherein the determining of the probabilities comprises calculating a plurality of preliminary probabilities for each of the data class candidates using different calculation methods. 
     
     
       13. The method of  claim 12 , wherein the determining of the probabilities further comprises calculating an averaged probability of the plurality of preliminary probabilities for each of the data class candidates. 
     
     
       14. The method of  claim 12 , wherein the determining of the probabilities further comprises:
 determining for the first data fields the data class candidate with a largest preliminary probability for the different calculation method; and 
 selecting for determining the probability for each of the first data fields the data class candidate for which most of the different calculation methods determined the largest preliminary probability. 
 
     
     
       15. A computer program product comprising a non-transitory-computer-readable storage medium having machine-executable program instructions embodied therewith for automatically classifying data fields of a dataset, execution of the program instructions by a processor of a computer system causing the processor to control the computer system to:
 receive training datasets, each training dataset comprising one or more training first data fields and previous user-selected data class assignments for the one or more training first data fields, each training first data field being assigned with a plurality of data class candidates; and 
 execute a machine learning algorithm on the training datasets to generate a trained machine learning model, wherein the learning algorithm is configured for:
 calculating, for each training dataset, clusters of data classes assigned to the data fields comprised by the retrieved training datasets; and 
 calculating, using a sequential pattern algorithm of the machine learning algorithm, probabilities indicating which data classes to be found together in a same dataset; 
 comparing the calculated probabilities to the retrieved user-selected data class assignments; 
 
 store confidence values for a plurality of data classes, wherein the confidence values are determined by applying a classifier to data fields of the dataset, wherein the classifier configured for determining the confidence values independently from one another for the data fields, the confidence value identifying a level of confidence that the respective data field belongs to the respective data class, the data class for which the confidence value exceeding a predefined threshold being identified as a data class candidate for the respective data field for which the respective confidence value is determined; 
 calculate first data fields of the dataset for which a single data class candidate being identifiable; 
 store a classification of the first data fields, wherein the classification of the first data fields comprises assigning the first data fields to the single data class identified as the data class candidate for the respective first data field; 
 calculate second data fields for which the plurality of data class candidates being identifiable; 
 calculate, by the trained machine learning model trained using previous user-selected data class assignments, a probability for the data class candidates identified for the second data fields that the respective data class candidate is a data class to which the respective second data field is to be assigned; 
 store a classification of the second data fields, wherein the classification is calculated, by a computer processor using the probabilities determined for the data class candidates by the trained machine learning model to select for the second data fields a data class from the data class candidates for the respective second data field to which the respective second data field being assigned; and 
 storing, in the computer readable storage medium, the dataset with metadata identifying for the classified data fields of the dataset the data classes to which the respective classified data fields are assigned; 
 import the classified dataset into a target database in the computer readable storage medium; 
 organize the target database using the target data model defining a class-based arrangement of data fields; 
 rearrange data fields of the dataset according to the target data model using metadata identifying data classes to which the data fields of the dataset are assigned; 
 add the rearranged dataset to the target database in accordance with the target data model; 
 execute a search query on the classified dataset with the metadata identifying data classes using a data class identifier as a search parameter of the search query; and 
 provide a search result, for the search query, comprise one or more data values comprised by datasets from data fields assigned to the data classes identified by the data class identifiers. 
 
     
     
       16. A computer system for automatically classifying data fields of a dataset, the computer system comprising:
 a processor and a memory storing machine-executable program instructions, execution of the program instructions by the processor causing the processor to control a computer system to:
 receive training datasets, each training dataset comprising one or more training first data fields and previous user-selected data class assignments for the one or more training first data fields, each training first data field being assigned with a plurality of data class candidates; and 
 execute a machine learning algorithm on the training datasets to generate a trained machine learning model, wherein executing the machine learning algorithm comprises:
 calculating, for each training dataset, clusters of data classes assigned to the data fields comprised by the retrieved training datasets; and 
 calculating, using a sequential pattern algorithm of the machine learning algorithm, probabilities indicating which data classes to be found together in a same dataset; 
 comparing the calculated probabilities to the retrieved user-selected data class assignments; 
 
 
 store, in the memory, confidence values for a plurality of data classes, wherein the confidence values are determined by applying a classifier to data fields of the dataset, wherein the classifier configured for determining the confidence values independently from one another for the data fields, the confidence value identifying a level of confidence that the respective data field belongs to the respective data class, the data class for which the confidence value exceeding a predefined threshold being identified as a data class candidate for the respective data field for which the respective confidence value is determined; 
 calculate first data fields of the dataset for which a single data class candidate being identifiable; 
 store, in the memory, a classification of the first data fields, wherein the classification of the first data fields comprises assigning the first data fields to the single data class identified as the data class candidate for the respective first data field; 
 calculate second data fields for which a plurality of data class candidates being identifiable; 
 calculate, by the trained machine learning model trained using previous user-selected data class assignments, a probability for the data class candidates identified for the second data fields that the respective data class candidate is a data class to which the respective second data field is to be assigned; 
 store, in the memory, a classification of the second data fields, wherein the classification is calculated, by a computer processor using the probabilities determined for the data class candidates by the trained machine learning model to select for the second data fields a data class from the data class candidates for the respective second data field to which the respective second data field being assigned; and 
 store, in a computer readable storage medium communicatively coupled to the processor, the dataset with metadata identifying for the classified data fields of the dataset the data classes to which the respective classified data fields are assigned; 
 import the classified dataset into a target database in the computer readable storage medium; 
 organize the target database using the target data model defining a class-based arrangement of data fields; 
 rearrange data fields of the dataset according to the target data model using metadata identifying data classes to which the data fields of the dataset are assigned; 
 add the rearranged dataset to the target database in accordance with the target data model; 
 execute a search query on the classified dataset with the metadata identifying data classes using a data class identifier as a search parameter of the search query; and 
 provide a search result, for the search query, comprise one or more data values comprised by datasets from data fields assigned to the data classes identified by the data class identifiers.