Prediction characterization for black box machine learning models

Abstract

A non-transitory computer-readable medium including instructions, which when executed by one or more processors of a computing system, causes the computing system to: access a machine learning model m, an input data point P to m, P including one or more features, and a prediction m(P) of m for P; create a set of perturbed input data points Pk from P by selecting a new value for at least one feature of P for each perturbed input data point; obtain a prediction m(Pk) for each of the perturbed input data points; analyze the predictions m(Pk) for the perturbed input data points to determine which features are most influential to the prediction; and output the analysis results to a user.

What is claimed is: 
     
         1 . One or more non-transitory computer-readable media including instructions, which when executed by one or more processors of a computing system, causes the computing system to:
 access a machine learning model m, an input data point P to m, P including one or more features, and a prediction m(P) of m for P;   create a set of perturbed input data points (Pk) from P by changing the value of at least one feature of P for each perturbed input data point;   obtain a prediction m(Pk) for each of the perturbed input data points;   analyze the predictions m(Pk) for the perturbed input data points to determine which features are most influential to the prediction; and   output the analysis results to a user.   
     
     
         2 . The one or more computer readable media of  claim 1 , wherein the values of multiple features of P are changed for at least some of the perturbed data points. 
     
     
         3 . The one or more computer readable media of  claim 1 , wherein analyze the predictions further includes to infer rules that distinguish those ones of the perturbed data points whose prediction m(Pk) is different than m(P) from those whose m(Pk) is the same as m(P). 
     
     
         4 . The one or more computer readable media of  claim 1 , wherein analyze the predictions further includes to infer rules that distinguish those ones of the perturbed data points whose prediction m(Pk) is different than m(P) from those whose m(Pk) is within a pre-defined distance of m(P). 
     
     
         5 . The one or more computer readable media of  claim 1 , wherein the pre-defined distance is one of Euclidean, L_1 norm, max_norm, or KL-divergence. 
     
     
         6 . The one or more computer readable media of  claim 4 , wherein the rules include a threshold above or below which the prediction m(Pk) changes from m(P). 
     
     
         7 . The one or more computer readable media of  claim 1 , wherein create a set of perturbed data points further includes to operate on P using a pre-defined perturbation function D. 
     
     
         8 . The one or more computer readable media of  claim 7 , D takes as input a single point P and outputs a set of n *perturbed* points {Pk_1, Pk_2, . . . Pk_n}. 
     
     
         9 . The one or more computer readable media of  claim 1 , further comprising instructions that, when executed, cause the computing system to access a histogram for each of the features of P, that indicates which bin of the histogram the value of each of the features in P falls within. 
     
     
         10 . The one or more computer readable media of  claim 9 , wherein changing the value of at least one feature of P for each perturbed input data point Pk includes selecting a new value for that feature from a different bin of the histogram. 
     
     
         11 . The one or more computer readable media of  claim 1 , further comprising instructions that, when executed, cause the computing system to display a prediction explanation graphic for either P or a Pk, to the user, the prediction explanation graphic including each of the features of the input data point, the value of each of those features, and a relative importance indication of each feature. 
     
     
         12 . The one or more computer readable media of  claim 1 , wherein the model is proprietary, accessible via a remote server, and further comprising instructions that, when executed, cause the computing system to access the model over a network link between the computing system and the remote server. 
     
     
         13 . A computing system comprising:
 one or more processors to implement a model characterization engine, the model characterization engine to:
 access a machine learning model m, an input data point P to m, P including one or more features, and a prediction m(P) of m for P; 
 create a set of perturbed input data points (Pk) from P by changing the value of at least one feature of P for each perturbed input data point; 
 obtain a prediction m(Pk) for each of the perturbed input data points; 
 analyze the predictions m(Pk) for the perturbed input data points to determine which features are most influential to the prediction; and 
 output the analysis results to a user. 
   
     
     
         14 . The computing system of  claim 13 , wherein the model characterization engine is to access the model from a remote server by inputting P and Pk to it, and receiving m(P) and m(Pk) from it, over a network link. 
     
     
         15 . The computing system of  claim 13 , wherein the model characterization engine creates the set of perturbed data points by operating on P using a pre-defined perturbation function D that takes as input a single point P and outputs a set of n *perturbed* points {Pk_1, Pk_2, . . . Pk_n}. 
     
     
         16 . The computing system of  claim 13 , wherein to analyze the predictions the model characterization engine is further to infer rules that distinguish those ones of the perturbed data points whose prediction m(Pk) is different than m(P) from those whose m(Pk) is the same as m(P). 
     
     
         17 . The computing system of  claim 13 , wherein to analyze the predictions the model characterization engine is further to infer rules that distinguish those ones of the perturbed data points whose prediction m(Pk) is different than m(P) from those whose m(Pk) is within a pre-defined distance of m(P). 
     
     
         18 . The computing system of  claim 17 , wherein the pre-defined distance is one of Euclidean, L_1 norm, max_norm, or KL-divergence. 
     
     
         19 . The computing system of  claim 16 , wherein the rules include a threshold above or below which the prediction m(Pk) changes from m(P), or changes at least by a pre-defined distance from m(P). 
     
     
         20 . The computing system of  claim 13 , wherein to output the analysis results to the user, the model characterization engine is further to cause the computing system to display a prediction explanation graphic for either P or a Pk to the user, the prediction explanation graphic including each of the features of P or Pk, as the case may be, the value of each of those features, and a relative importance indication of each feature.