Systems and methods for detecting personally identifiable information

Abstract

The disclosure generally relates to systems and methods for detecting personally identifiable information (PII). The present systems and methods solve the problem of detecting the PII and the PII column names in the customer database with enhanced accuracy, by developing a PII classification model trained with an enhanced and effective training dataset. An enhanced sub-metadata from the metadata having the plurality of the column names is obtained by using highest match distance values, the string comparator values, and the is PII indicator values. The enhanced sub-metadata comprising the column names that can be easily differentiated as PII columns or non-PII columns. Hence the training dataset and the testing dataset obtained from the enhanced sub-metadata improves the accuracy of the PII classification model. Preventive measures can be taken to protect such detected PII present under the PII columns by employing various data privacy and protection techniques.

What is claimed is: 
     
       1. A processor-implemented method, comprising the steps of:
 receiving, via one or more hardware processors, a defined personally identifiable information (PII) dataset, wherein the defined PII dataset comprises a plurality of PII column names; 
 receiving, via the one or more hardware processors, a metadata comprising a plurality of column names from a data source, wherein each column name of the plurality of column names is indicative of one of a PII category and a non-PII category; 
 calculating, via the one or more hardware processors, a highest match distance value for each column name of the plurality of column names present in the metadata, based on the plurality of PII column names present in the defined PII dataset, wherein the highest match distance value is in terms of percentage (‘%’); 
 determining, via the one or more hardware processors, a string comparator value for each column name of the plurality of column names present in the metadata, based on the plurality of PII column names present in the defined PII dataset; 
 evaluating, via the one or more hardware processors, an isPII indicator value for each column name of the plurality of column names present in the metadata, based on the highest match distance value and the string comparator value; 
 obtaining, via the one or more hardware processors, a sub-metadata comprising one or more column names selected from the plurality of column names present in the metadata, based on the highest match distance value and the isPII indicator value; 
 dividing, via the one or more hardware processors, the one or more column names of the sub-metadata into a training dataset and a testing dataset, based on a random sampling method; 
 generating, via the one or more hardware processors, a training sparse matrix based on the training dataset and the sub-metadata, and a testing sparse matrix based on the testing dataset and the sub-metadata; 
 obtaining, via the one or more hardware processors, training sparse matrix features and testing sparse matrix features, wherein the training sparse matrix features are indicative of matrix elements present in the training sparse matrix and the testing sparse matrix features are indicative of matrix elements present in the testing sparse matrix; and 
 generating, via the one or more hardware processors, a PII classification model, by training a machine learning model with an input training dataset, wherein the input training dataset comprises the training sparse matrix features, associated highest match distance values, and associated isPII indicator values, wherein the PII classification model calculates a PII score and a non-PII score for each column name to be detected as the PII column or the non-PII column using the input sparse matrix features and the highest match distance value of the corresponding column name, wherein the machine learning model learns a relation between the PII score and the non-PII score and correlates with the isPII indicator value of the corresponding column name, 
 wherein the PII score is computed with a probability score of the column name based on presence as PII in an input sparse matrix, a probability of matching percentage indicating the probability score with reference to the highest match distance value of the column name, a probability score of the column name being the PII column that is calculated based on the isPII indicator value, and a Laplace smooth value which is considered as ‘1’, 
 wherein the non-PII score is computed with a probability score of the column name based on presence as non-PII in the input sparse matrix, a probability of the matching percentage indicating the probability score with reference to the highest match distance value of the column name, a probability score of the column name being the non-PII column that is calculated based on the isPII indicator value, and a Laplace smooth value which is considered as ‘1’, 
 wherein during training of the machine learning model, when the PII score of the column name is greater than the non-PII score, the column name is said to be PII column name and the machine learning model learns the associated is isPII indicator value as ‘1’ and when the PII score of the column name is lesser than the non-PII score, the column name is said to be the non-PII and the machine learning model learns the associated isPII indicator value as ‘0’. 
 
     
     
       2. The method as claimed in  claim 1 , further comprising, validating the PII classification model, using an input testing dataset comprising testing sparse matrix features, the associated highest match distance values, and the associated isPII indicator values. 
     
     
       3. The method as claimed in  claim 1 , further comprising:
 receiving, via the one or more hardware processors, an input customer database comprising one or more input column names to be detected as PII column or non-PII column; calculating, via the one or more hardware processors, the highest match distance value for each input column name of the one or more input column names present in the input customer database; 
 generating, via the one or more hardware processors, an input sparse matrix based on the input customer database and the sub-metadata; 
 obtaining, via the one or more hardware processors, input sparse matrix features that are indicative of matrix elements present in the input sparse matrix; and 
 detecting, via the one or more hardware processors, each input column name of the one or more input column names as the PII column or the non-PI column, using the PII classification model, based on the input sparse matrix features and the associated highest match distance value, wherein the PII classification model detects the input column name as the PII column, when the PII score is greater than the non-PII score, wherein the PII classification model detects the corresponding input column name as the non-PII column, when the PII score is less than the non-PII score. 
 
     
     
       4. The method as claimed in  claim 1 , wherein determining the highest match distance value for each column name of the plurality of column names present in the metadata, comprises:
 calculating a set of match distance values for each column name, based on a match distance value between a corresponding column name and each PII column name of the plurality of PII column names present in the defined PII dataset; and 
 determining a highest value among the set of match distance values, as the highest match distance value for the corresponding column name. 
 
     
     
       5. The method as claimed in  claim 1 , wherein the string comparator value for each column name is determined by comparing corresponding column name with each PII column name of the plurality of PII column names present in the defined PII dataset. 
     
     
       6. The method as claimed in  claim 1 , wherein the training sparse matrix is generated based on the one or more column names present in the training dataset and one or more unique words obtained from the one or more column names present in the sub-metadata. 
     
     
       7. The method as claimed in  claim 1 , wherein the testing sparse matrix is generated based on the one or more column names present in the testing dataset and the one or more unique words obtained from the one or more column names present in the sub-metadata. 
     
     
       8. The method as claimed in  claim 3 , wherein determining the highest match distance value for each input column name of the one or more input column names present in the input customer database, comprises:
 calculating a set of match distance values for each input column name, based on the match distance value between a corresponding input column name and each PII column name of the plurality of PII column names present in the defined PII dataset; and 
 determining a highest value among the set of match distance values, as the highest match distance value for the corresponding input column name. 
 
     
     
       9. The method as claimed in  claim 3 , wherein the input sparse matrix is generated based on the one or more input column names present in the input customer database, and the one or more unique words obtained from the one or more column names present in the sub-metadata. 
     
     
       10. A system comprising:
 one or more data storage devices operatively coupled to one or more hardware processors and configured to store instructions which when executed cause the one or more hardware processors to: 
 receive a defined personally identifiable information (PII) dataset, wherein the defined PII dataset comprises a plurality of PII column names; 
 receive a metadata comprising a plurality of column names from a data source, wherein each column name of the plurality of column names is indicative of one of a PII category and a non-PII category; 
 calculate a highest match distance value for each column name of the plurality of column names present in the metadata, based on the plurality of PII column names present in the defined PII dataset, wherein the highest match distance value is in terms of percentage (‘%’); 
 determine a string comparator value for each column name of the plurality of column names present in the metadata, based on the plurality of PII column names present in the defined PII dataset; 
 evaluate an isPII indicator value for each column name of the plurality of column names present in the metadata, based on the highest match distance value and the string comparator value; 
 obtain a sub-metadata comprising one or more column names selected from the plurality of column names present in the metadata, based on the highest match distance value and the isPII indicator value; 
 divide the one or more column names of the sub-metadata into a training dataset and a testing dataset, based on a random sampling method; 
 generate a training sparse matrix based on the training dataset and the sub-metadata, and a testing sparse matrix based on the testing dataset and the sub-metadata; 
 obtain training sparse matrix features and testing sparse matrix features, wherein the training sparse matrix features are indicative of matrix elements present in the training sparse matrix and the testing sparse matrix features are indicative of matrix elements present in the testing sparse matrix; and 
 generate a PII classification model, by training a machine learning model with an input training dataset, wherein the input training dataset comprises the training sparse matrix features, associated highest match distance values, and associated isPII indicator values, wherein the PII classification model calculates a PII score and a non-PII score for each column name to be detected as the PII column or the non-PII column using the input sparse matrix features and the highest match distance value of the corresponding column name, wherein the machine learning model learns a relation between the PII score and the non-PII score and correlates with the isPII indicator value of the corresponding column name, 
 wherein the PII score is computed with a probability score of the column name based on presence as PII in an input sparse matrix, a probability of matching percentage indicating the probability score with reference to the highest match distance value of the column name, a probability score of the column name being the PII column that is calculated based on the isPII indicator value, and a Laplace smooth value which is considered as ‘1’, 
 wherein the non-PII score is computed with a probability score of the column name based on presence as non-PII in the input sparse matrix, a probability of the matching percentage indicating the probability score with reference to the highest match distance value of the column name, a probability score of the column name being the non-PII column that is calculated based on the isPII indicator value, and a Laplace smooth value which is considered as ‘1’, 
 wherein during training of the machine learning model, when the PII score of the column name is greater than the non-PII score, the column name is said to be PII column name and the machine learning model learns the associated isPII indicator value as ‘1’ and when the PII score of the column name is lesser than the non-PII score, then the column name is said to be the non-PII and the machine learning model learns the associated isPII indicator value as ‘0’. 
 
     
     
       11. The system as claimed in  claim 10 , wherein the one or more hardware processors are further configured by the instructions to validate the PII classification model, using an input testing dataset comprising testing sparse matrix features, the associated highest match distance values, and the associated isPII indicator values. 
     
     
       12. The system as claimed in  claim 10 , wherein the one or more hardware processors are further configured by the instructions to:
 receive an input customer database comprising one or more input column names to be detected as PII column or non-PII column; 
 calculate the highest match distance value for each input column name of the one or more input column names present in the input customer database; 
 generate an input sparse matrix based on the input customer database and the sub-metadata; 
 obtain input sparse matrix features that are indicative of matrix elements present in the input sparse matrix; and 
 detect each input column name of the one or more input column names as the PII column or the non-PII column, using the PII classification model, based on the input sparse matrix features and the associated highest match distance value, wherein the PII classification model detects the input column name as the PII column, when the PII score is greater than the non-PII score, wherein the PII classification model detects the corresponding input column name as the non-PII column, when the PII score is less than the non-PII score. 
 
     
     
       13. The system as claimed in  claim 10 , wherein the one or more hardware processors are further configured by the instructions to determine the highest match distance value for each column name of the plurality of column names present in the metadata, by:
 calculating a set of match distance values for each column name, based on a match distance value between a corresponding column name and each PII column name of the plurality of PII column names present in the defined PII dataset; and 
 determining a highest value among the set of match distance values, as the highest match distance value for the corresponding column name. 
 
     
     
       14. The system as claimed in  claim 10 , wherein the one or more hardware processors are further configured by the instructions to determine the string comparator value for each column name, by comparing corresponding column name with each PII column name of the plurality of PII column names present in the defined PII dataset. 
     
     
       15. The system as claimed in  claim 10 , wherein the one or more hardware processors are further configured by the instructions to generate the training sparse matrix, based on the one or more column names present in the training dataset and one or more unique words obtained from the one or more column names present in the sub-metadata. 
     
     
       16. The system as claimed in  claim 10 , wherein the one or more hardware processors are further configured by the instructions to generate the testing sparse matrix, based on the one or more column names present in the testing dataset and the one or more unique words obtained from the one or more column names present in the sub-metadata. 
     
     
       17. The system as claimed in  claim 12 , wherein the one or more hardware processors are further configured by the instructions to determine the highest match distance value for each input column name of the one or more input column names present in the input customer database, by:
 calculating a set of match distance values for each input column name, based on the match distance value between a corresponding input column name and each PII column name of the plurality of PII column names present in the defined PII dataset; and 
 determining a highest value among the set of match distance values, as the highest match distance value for the corresponding input column name. 
 
     
     
       18. The system as claimed in  claim 12 , wherein the one or more hardware processors are further configured by the instructions to generate the input sparse matrix, based on the one or more input column names present in the input customer database, and the one or more unique words obtained from the one or more column names present in the sub-metadata. 
     
     
       19. A computer program product comprising a non-transitory computer readable medium having a computer readable program embodied therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:
 receive a defined personally identifiable information (PII) dataset, wherein the defined PII dataset comprises a plurality of PII column names; 
 receive a metadata comprising a plurality of column names from a data source, wherein each column name of the plurality of column names is indicative of one of a PII category and a non-PII category; 
 calculate a highest match distance value for each column name of the plurality of column names present in the metadata, based on the plurality of PII column names present in the defined PII dataset, wherein the highest match distance value is in terms of percentage (‘%’); 
 determine a string comparator value for each column name of the plurality of column names present in the metadata, based on the plurality of PII column names present in the defined PII dataset; 
 evaluate an isPII indicator value for each column name of the plurality of column names present in the metadata, based on the highest match distance value and the string comparator value; 
 obtain a sub-metadata comprising one or more column names selected from the plurality of column names present in the metadata, based on the highest match distance value and the isPII indicator value; 
 divide the one or more column names of the sub-metadata into a training dataset and a testing dataset, based on a random sampling method; 
 generate a training sparse matrix based on the training dataset and the sub-metadata, and a testing sparse matrix based on the testing dataset and the sub-metadata; 
 obtain training sparse matrix features and testing sparse matrix features, wherein the training sparse matrix features are indicative of matrix elements present in the training Sparse matrix and the testing sparse matrix features are indicative of matrix elements present in the testing sparse matrix; and 
 generate a PII classification model, by training a machine learning model with an input training dataset, wherein the input training dataset comprises the training sparse matrix features, associated highest match distance values, and associated isPII indicator values, wherein the PII classification model calculates a PII score and a non-PII score for each column name to be detected as the PII column or the non-PII column using the input sparse matrix features and the highest match distance value of the corresponding column name, wherein the machine learning model learns a relation between the PII score and the non-PII score and correlates with the isPII indicator value of the corresponding column name, 
 wherein the PII score is computed with a probability score of the column name based on presence as PII in an input sparse matrix, a probability of matching percentage indicating the probability score with reference to the highest match distance value of the column name, a probability score of the column name being the PII column that is calculated based on the isPII indicator value, and a Laplace smooth value which is considered as ‘1’, 
 wherein the non-PII score is computed with a probability score of the column name based on presence as non-PII in the input sparse matrix, a probability of the matching percentage indicating the probability score with reference to the highest match distance value of the column name, a probability score of the column name being the non-PII column that is calculated based on the isPII indicator value, and a Laplace smooth value which is considered as ‘1’, 
 wherein during training of the machine learning model, when the PII score of the column name is greater than the non-PII score, the column name is said to be PII column name and the machine learning model learns the associated isPII indicator value as ‘1’ and when the PII score of the column name is lesser than the non-PII score, the column name is said to be the non-PII and the machine learning model learns the associated isPII indicator value as ‘0’.