Method for analyzing a prediction classification in a machine learning model

Abstract

A method is provided for analyzing a classification in a machine learning model (ML). In the method, the ML model is trained using a training dataset to produce a trained ML model. One or more samples are provided to the trained ML model to produce one or more prediction classifications. A gradient is determined for the one of more samples at a predetermined layer of the trained ML model. The one or more gradients and the one or more prediction classifications for each sample are stored. Also, an intermediate value of the ML model may be stored. Then, a sample is chosen to analyze. A gradient of the sample is determined if the gradient was not already determined when the at least one gradient is determined. Using the at least one gradient, and one or more of a data structure, a predetermined metric, and an intermediate value, the k nearest neighbors to the sample are determined. A report comprising the sample and the k nearest neighbors may be provided for analysis.

What is claimed is: 
     
       1. A method for analyzing a classification in a machine learning model (ML), the method comprising:
 training the ML model using a training dataset having a plurality of samples to produce a trained ML model; 
 inputting one or more samples to the trained ML model to produce one or more prediction classifications; 
 determining a gradient of the one or more samples at a predetermined layer of the trained ML model; 
 storing the one or more gradients and the one or more prediction classifications for each of the one or more samples; 
 choosing a sample to analyze, wherein a gradient of the sample is determined if the gradient was not already determined when the gradient of the one or more samples is determined; 
 determining, using the at least one gradient, a data structure, and predetermined metric, k nearest neighbors to the sample, where k is an integer; and 
 generating a report comprising the sample and the k nearest neighbors. 
 
     
     
       2. The method of  claim 1 , wherein the data structure comprises a kNN, Kd-tree, or R-Tree data structure. 
     
     
       3. The method of  claim 1 , wherein the at least one gradient is calculated using one or more of the weights or biases of the predetermined layer. 
     
     
       4. The method of  claim 1 , wherein the predetermined metric is a distance metric comprising one or more of a Manhattan distance, a Euclidean distance, or a hamming distance. 
     
     
       5. The method of  claim 1 , wherein choosing a sample to analyze further comprises choosing a sample that is misclassified by the trained ML model. 
     
     
       6. The method of  claim 1 , wherein determining a gradient for the one or more samples at a predetermined layer of the trained ML model further comprises determining a gradient for the one or more samples at a last convolutional intermediate layer. 
     
     
       7. The method of  claim 1 , wherein generating a report further comprises generating a report using Grad-CAM (gradient class-activation map). 
     
     
       8. The method of  claim 1 , wherein the k nearest neighbors are the k nearest misclassified samples that share the same misclassification. 
     
     
       9. The method of  claim 1 , wherein the k nearest neighbors are the k nearest samples having various different misclassifications. 
     
     
       10. The method of  claim 1 , further comprising combining the gradient of the one or more samples at the predetermined layer with an intermediate value of the predetermined layer. 
     
     
       11. The method of  claim 1 , wherein choosing a sample to analyze further comprises choosing a sample from the plurality of samples. 
     
     
       12. A method for analyzing a classification in a machine learning model (ML), the method comprising:
 training the ML model using a training dataset having a plurality of samples to produce a trained ML model; 
 inputting one or more samples to the trained ML model to produce one or more predicted classifications; 
 determining a gradient for the one or more samples at a predetermined layer of the trained ML model to produce a plurality of gradients; 
 storing the one or more gradients and the one or more predicted classifications for each of the one or more samples; 
 choosing a sample to analyze, wherein a gradient of the sample is determined if the gradient of the sample chosen to for analysis is not already determined; 
 determining, using the one or more stored gradients, a data structure, and a distance metric, k nearest neighbors to the sample, where k is an integer, and wherein the k nearest neighbors comprise misclassified samples of the sample being analyzed; and 
 generating a report comprising the sample, the distance metric, and the k nearest neighbors. 
 
     
     
       13. The method of  claim 12 , wherein the data structure comprises a kNN, Kd-tree, or R-Tree data structure. 
     
     
       14. The method of  claim 12 , wherein the ML model includes a neural network. 
     
     
       15. The method of  claim 12 , wherein the predetermined metric is a distance metric comprising one or more of a Manhattan distance, a Euclidean distance, or a hamming distance. 
     
     
       16. The method of  claim 12 , wherein choosing a sample to analyze further comprises choosing a sample that is misclassified by the trained ML model. 
     
     
       17. The method of  claim 12 , wherein determining a gradient for the one or more samples at a predetermined layer of the trained ML model further comprises determining a gradient for the one or more samples at a last convolutional layer. 
     
     
       18. The method of  claim 12 , further comprising combining the gradient of the one or more samples at the predetermined layer with an intermediate value of the predetermined layer. 
     
     
       19. The method of  claim 12 , wherein the k nearest neighbors are the k nearest misclassified samples that share the same misclassification. 
     
     
       20. The method of  claim 12 , wherein choosing a sample to analyze further comprises choosing a sample from the plurality of samples.