System and method for improving machine learning models by detecting and removing inaccurate training data

Abstract

Methods, systems and computer program products are described to improve machine learning (ML) model-based classification of data items by identifying and removing inaccurate training data. Inaccurate training samples may be identified, for example, based on excessive variance in vector space between a training sample and a mean of category training samples, and based on a variance between an assigned category and a predicted category for a training sample. Suspect or erroneous samples may be selectively removed based on, for example, vector space variance and/or prediction confidence level. As a result, ML model accuracy may be improved by training on a more accurate revised training set. ML model accuracy may (e.g., also) be improved, for example, by identifying and removing suspect categories with excessive (e.g., weighted) vector space variance. Suspect categories may be retained or revised. Users may (e.g., also) specify a prediction confidence level and/or coverage (e.g., to control accuracy).

What is claimed is: 
     
       1. A method, comprising:
 receiving a training set comprising training samples, including a first training sample, associated or labeled with a first category to train a first machine learning (ML) model to predict whether an inference sample is associated with the first category, where a sample comprises a vectorized data item; and 
 improving prediction accuracy of the first ML model by identifying the first training sample as an erroneous sample based on: 
 a first variance in vector space between the first training sample and a category mean comprising an average of the training samples; and 
 a second variance between the first category and a predicted category for the first training sample. 
 
     
     
       2. The method of  claim 1 , wherein the second variance comprises:
 the predicted category as a second category, different from the first category, and a confidence level in an accuracy of the predicted category. 
 
     
     
       3. The method of  claim 2 , wherein the identification of the first training sample as an erroneous sample comprises:
 determining that the first variance exceeds a first threshold; and 
 determining that the confidence level exceeds a second threshold. 
 
     
     
       4. The method of  claim 3 , further comprising:
 identifying the first training sample as a suspect sample based on one of: 
 determining that the first variance exceeds the first threshold; or 
 determining that the confidence level exceeds the second threshold. 
 
     
     
       5. The method of  claim 3 , further comprising:
 creating a removal list by ordering identified erroneous samples from a highest to a lowest first variance; and 
 selectively removing the erroneous samples in order from the removal list to create a revised training set. 
 
     
     
       6. The method of  claim 5 , further comprising:
 receiving a user input indicating whether or how many erroneous samples to remove. 
 
     
     
       7. The method of  claim 5 , further comprising:
 training the first ML model with the revised training set instead of the training set. 
 
     
     
       8. The method of  claim 4 , further comprising:
 creating a suspect removal list by ordering suspect samples from a lowest to a highest confidence level in the accuracy of the predicted category; and 
 selectively removing the suspect samples in order from the removal list to create a revised training set. 
 
     
     
       9. The method of  claim 1 , wherein a second ML model predicts the predicted category. 
     
     
       10. The method of  claim 1 , further comprising:
 receiving a first user selection indicating coverage or a second user selection indicating a confidence level threshold for the first ML model to categorize inference samples; 
 generating, in response to the first user selection, a corresponding confidence level threshold; 
 generating, in response to the second user selection, a corresponding coverage; 
 predicting a category for each of the inference samples and associating with each predicted category a prediction confidence level; 
 categorizing an inference sample in the first category if the prediction confidence level is above the selected or the corresponding confidence level threshold, and not categorizing an inference sample in the first category when the prediction confidence level is below the selected or the corresponding confidence level threshold. 
 
     
     
       11. The method of  claim 1 , further comprising:
 generating a category score for the first category based on a total variance of the training samples associated or labeled with the first category; 
 identifying the first category as a suspect category subject to removal when the category score indicates excessive variance of the training samples relative to a threshold variance score. 
 
     
     
       12. The method of  claim 11 , wherein the total variance is calculated based on weighted elements of data items vectorized in the training samples. 
     
     
       13. The method of  claim 11 , further comprising:
 notifying a user about the suspect category; and 
 requesting the user to indicate whether to use or to modify the suspect category. 
 
     
     
       14. A system, comprising:
 at least one processor circuit; and 
 at least one memory that stores program code configured to be executed by the at least one processor circuit, the program code comprising: 
 a data fetcher configured to receive a training set comprising training samples, including a first training sample, associated or labeled with a first category to train a first machine learning (ML) model to predict whether an inference sample is associated with the first category, where a sample comprises a vectorized data item; and 
 an artificial intelligence (AI) engine configured to: 
 identify the first training sample as an erroneous sample based on: 
 a first variance in vector space between the first training sample and a category mean comprising an average of the training samples; and 
 a second variance between the first category and a predicted category for the first training sample. 
 
     
     
       15. The system of  claim 14 , the AI engine further configured to:
 selectively remove identified erroneous samples from the training set to create a revised training set; and 
 train the first ML model with the revised training set instead of the training set. 
 
     
     
       16. The system of  claim 15 , the AI engine further configured to:
 receive a first user selection indicating coverage or a second user selection indicating a confidence level threshold for the first ML model to categorize inference samples; 
 select, in response to the first user selection, a corresponding confidence level threshold; 
 select, in response to the second user selection, a corresponding coverage; 
 predict a category for each of the inference samples and associating with each predicted category a prediction confidence level; and 
 categorize the inference sample in the first category if the prediction confidence level is above the selected or the corresponding confidence level threshold and not categorizing the inference sample in the first category if the prediction confidence level is below the selected or the corresponding confidence level threshold. 
 
     
     
       17. The system of  claim 14 , the AI engine further configured to:
 generate a category score for the first category based on a total variance of the training samples associated or labeled with the first category; 
 identify the first category as a suspect category subject to removal when the category score indicates excessive variance of the training samples relative to a threshold variance score; and 
 request user input to indicate whether to use or to modify the suspect category. 
 
     
     
       18. A computer-readable storage medium having program instructions recorded thereon that, when executed by at least one processor of a computing device, perform a method, the method comprising:
 receiving a training set comprising training samples, including a first training sample, associated or labeled with a first category to train a first machine learning (ML) model to predict whether an inference sample is associated with the first category, where a sample comprises a vectorized data item; and 
 improving prediction accuracy of the first ML model by identifying the first training sample as an erroneous sample based on: 
 a first variance in vector space between the first training sample and a category mean comprising an average of the training samples; and 
 a second variance between the first category and a predicted category for the first training sample, wherein the second variance comprises the predicted category as a second category, different from the first category, and a confidence level in an accuracy of the predicted category; 
 selectively removing at least one identified erroneous sample from the training set to create a revised training set; and 
 training the first ML model with the revised training set instead of the training set. 
 
     
     
       19. The computer-readable storage medium of  claim 18 , the method further comprising:
 receiving at least one of a first user selection indicating coverage or a second user selection indicating a confidence level threshold for the first ML model to categorize inference samples; 
 selecting, in response to the first user selection, a corresponding confidence level threshold; 
 selecting, in response to the second user selection, a corresponding coverage; 
 predicting a category for each of the inference samples and associating with each predicted category a prediction confidence level; and 
 categorizing an inference sample in the first category if the prediction confidence level is above the selected or the corresponding confidence level threshold and not categorizing an inference sample in the first category if the prediction confidence level is below the selected or the corresponding confidence level threshold. 
 
     
     
       20. The computer-readable storage medium of  claim 18 , the method further comprising:
 generating a category score for the first category based on a total variance of the training samples associated or labeled with the first category; and 
 
       identifying the first category as a suspect category subject to removal when the category score indicates excessive variance of the training samples relative to a threshold variance.