Predictability-driven compression of training data sets

Abstract

Techniques for performing predictability-driven compression of training data sets used for machine learning (ML) are provided. In one set of embodiments, a computer system can receive a training data set comprising a plurality of data instances and can train an ML model using the plurality of data instances, the training resulting in a trained version of the ML model. The computer system can further generate prediction metadata for each data instance in the plurality of data instances using the trained version of the ML model and can compute a predictability measure for each data instance based on the prediction metadata, the predictability measure indicating a training value of the data instance. The computer system can then filter one or more data instances from the plurality of data instances based on the computed predictability measures, the filtering resulting in a compressed version of the training data set.

What is claimed is: 
     
       1. A method comprising:
 receiving, by a computer system, a training data set comprising a plurality of data instances; 
 training, by the computer system, a first machine learning (ML) model using the plurality of data instances, the training resulting in a trained version of the first ML model; 
 for each data instance in the plurality of data instances:
 generating, by the computer system, prediction metadata for the data instance via the trained version of the first ML model; and 
 computing, by the computer system, a predictability measure for the data instance based on the prediction metadata, the predictability measure indicating a training value of the data instance; and 
 
 filtering, by the computer system, one or more data instances from the plurality of data instances based on the computed predictability measures, the filtering resulting in a compressed version of the training data set. 
 
     
     
       2. The method of  claim 1  further comprising:
 training a second ML model using the compressed version of the training data set, the second ML model being a more complex ML model than the first ML model. 
 
     
     
       3. The method of  claim 1  wherein the first ML model is an ML classifier and wherein the prediction metadata generated for each data instance includes a class distribution vector. 
     
     
       4. The method of  claim 3  wherein computing the predictability measure comprises:
 retrieving, from the training data set, a class label for the data instance; 
 constructing a perfect class distribution vector based on the class label; and 
 computing a distance between the class distribution vector and the perfect class distribution vector. 
 
     
     
       5. The method of  claim 1  wherein a high value for the predictability measure indicates that the data instance has low training value and wherein a low value for the predictability measure indicates that the data instance has high training value. 
     
     
       6. The method of  claim 1  wherein the filtering comprises:
 removing, from the training data set, data instances with high predictability measures at a higher frequency or likelihood than data instances with low predictability measures. 
 
     
     
       7. The method of  claim 2  further comprising:
 receiving a query data instance; 
 generating, via the trained version of the first ML model, a first prediction for the query data instance and a confidence level for the first prediction; 
 if the confidence level for the first prediction exceeds a threshold, returning the first prediction as a final prediction result for the query data instance; and 
 if the confidence level for the first prediction does not exceed the threshold:
 generating, via the trained version of the second ML model, a second prediction for the query data instance; and 
 returning the second prediction as the final prediction result for the query data instance. 
 
 
     
     
       8. A non-transitory computer readable storage medium having stored thereon program code executable by a computer system, the program code causing the computer system to execute a method comprising:
 receiving a training data set comprising a plurality of data instances; 
 training a first machine learning (ML) model using the plurality of data instances, the training resulting in a trained version of the first ML model; 
 for each data instance in the plurality of data instances:
 generating prediction metadata for the data instance via the trained version of the first ML model; and 
 computing a predictability measure for the data instance based on the prediction metadata, the predictability measure indicating a training value of the data instance; and 
 
 filtering one or more data instances from the plurality of data instance based on the computed predictability measures, the filtering resulting in a compressed version of the training data set. 
 
     
     
       9. The non-transitory computer readable storage medium of  claim 8  wherein the method further comprises:
 training a second ML model using the compressed version of the training data set, the second ML model being a more complex ML model than the first ML model. 
 
     
     
       10. The non-transitory computer readable storage medium of  claim 8  wherein the first ML model is an ML classifier and wherein the prediction metadata generated for each data instance includes a class distribution vector. 
     
     
       11. The non-transitory computer readable storage medium of  claim 10  wherein computing the predictability measure comprises:
 retrieving, from the training data set, a class label for the data instance; 
 constructing a perfect class distribution vector based on the class label; and 
 computing a distance between the class distribution vector and the perfect class distribution vector. 
 
     
     
       12. The non-transitory computer readable storage medium of  claim 8  wherein a high value for the predictability measure indicates that the data instance has low training value and wherein a low value for the predictability measure indicates that the data instance has high training value. 
     
     
       13. The non-transitory computer readable storage medium of  claim 8  wherein the filtering comprises:
 removing, from the training data set, data instances with high predictability measures at a higher frequency or likelihood than data instances with low predictability measures. 
 
     
     
       14. The non-transitory computer readable storage medium of  claim 9  wherein the method further comprises:
 receiving a query data instance; 
 generating, via the trained version of the first ML model, a first prediction for the query data instance and a confidence level for the first prediction; 
 if the confidence level for the first prediction exceeds a threshold, returning the first prediction as a final prediction result for the query data instance; and 
 if the confidence level for the first prediction does not exceed the threshold:
 generating, via the trained version of the second ML model, a second prediction for the query data instance; and 
 returning the second prediction as the final prediction result for the query data instance. 
 
 
     
     
       15. A computer system comprising:
 a processor; and 
 a non-transitory computer readable medium having stored thereon program code that, when executed, causes the processor to:
 receive a training data set comprising a plurality of data instances; 
 train a first machine learning (ML) model using the plurality of data instances, the training resulting in a trained version of the first ML model; 
 for each data instance in the plurality of data instances:
 generate prediction metadata for the data instance via the trained version of the first ML model; and 
 compute a predictability measure for the data instance based on the prediction metadata, the predictability measure indicating a training value of the data instance; and 
 
 filter one or more data instances from the plurality of data instances based on the computed predictability measures, the filtering resulting in a compressed version of the training data set. 
 
 
     
     
       16. The computer system of  claim 15  wherein the program code further causes the processor to:
 train a second ML model using the compressed version of the training data set, the second ML model being a more complex ML model than the first ML model. 
 
     
     
       17. The computer system of  claim 15  wherein the first ML model is an ML classifier and wherein the prediction metadata generated for each data instance includes a class distribution vector. 
     
     
       18. The computer system of  claim 17  wherein the program code that causes the processor to compute the predictability measure comprises program code that causes the processor to:
 retrieve, from the training data set, a class label for the data instance; 
 construct a perfect class distribution vector based on the class label; and 
 compute a distance between the class distribution vector and the perfect class distribution vector. 
 
     
     
       19. The computer system of  claim 15  wherein a high value for the predictability measure indicates that the data instance has low training value and wherein a low value for the predictability measure indicates that the data instance has high training value. 
     
     
       20. The computer system of  claim 15  wherein the program code that causes the processor to filter the one or more data instances comprises program code that causes the processor to:
 remove, from the training data set, data instances with high predictability measures at a higher frequency or likelihood than data instances with low predictability measures. 
 
     
     
       21. The computer system of  claim 16  wherein the program code further causes the processor to:
 receive a query data instance; 
 generate, via the trained version of the first ML model, a first prediction for the query data instance and a confidence level for the first prediction; 
 if the confidence level for the first prediction exceeds a threshold, return the first prediction as a final prediction result for the query data instance; and 
 if the confidence level for the first prediction does not exceed the threshold:
 generate, via the trained version of the second ML model, a second prediction for the query data instance; and 
 return the second prediction as the final prediction result for the query data instance.