Interface for visualizing and improving model performance

Abstract

Performance of a first generated model can be monitored while the first generated model is deployed for use on live data. The monitoring can include determining a first performance value of the first generated model. Performance of a second generated model can be monitored while the second generated model is deployed for use on live data. The monitoring can include determining a second performance value of the second generated model. A plot including a first axis and a second axis can be rendered. The first axis can include a characterization of a first performance metric and the second axis can include a characterization of a second performance metric. A first graphical object at a first location characterizing the first performance value and a second graphical object at a second location characterizing the second performance value can be rendered. Related apparatus, systems, techniques and articles are also described.

What is claimed is: 
     
         1 . A method comprising:
 monitoring performance of a first generated model while the first generated model is deployed for use on live data, the monitoring including determining a first performance value of the first generated model;   monitoring performance of a second generated model while the second generated model is deployed for use on live data, the monitoring including determining a second performance value of the second generated model;   rendering, within a graphical user interface, a plot including a first axis and a second axis, the first axis including a characterization of a first performance metric and the second axis including a characterization of a second performance metric; and   rendering, within the graphical user interface and the plot, a first graphical object at a first location characterizing the first performance value and a second graphical object at a second location characterizing the second performance value.   
     
     
         2 . The method of  claim 1 , wherein the first axis is indicative of a positive or a negative outcome; and wherein the second axis is indicative of a correct or incorrect prediction. 
     
     
         3 . The method of  claim 2 , wherein a size of the first graphical object is indicative of a scale of the first performance value. 
     
     
         4 . The method of  claim 2 , wherein a size of the first graphical object is indicative of a relative cost or a relative benefit. 
     
     
         5 . The method of  claim 1 , wherein the first performance value includes a count of a positive outcome or a count of a negative outcome, the first performance value further includes a count of a correct prediction or an incorrect prediction, the method further comprising:
 adjusting a ratio of outcomes according to a count of records per period.   
     
     
         6 . The method of  claim 1 , wherein the first generated model has been trained on historical data, the method further comprising:
 determining future cost or net benefit of the first deployed model over time.   
     
     
         7 . The method of  claim 6 , further comprising:
 rendering, within the graphical user interface, a characterization of the future cost or net benefit of the first deployed model over time.   
     
     
         8 . The method of  claim 1 , wherein the first generated model has been trained on historical data and wherein each transaction in the live data includes an associated characteristic, the method further comprising:
 determining future cost or net benefit of the first deployed model based on a change in distribution of transaction characteristics of the data source of the first model and over time.   
     
     
         9 . The method of  claim 8 , wherein the associated characteristic characterizes a specific subgroup of the population, the specific subgroup including a geographic location associated with a respective transaction, a component failure, or a capacity measure. 
     
     
         10 . The method of  claim 8 , wherein a distribution of transaction characteristics of the live data is different than a training distribution of transaction characteristics of the historical data. 
     
     
         11 . The method of  claim 1 , the method further comprising:
 identifying, within the live data, subgroups of the live data;   determining a performance metric for the first generated model and for each of the subgroups of the live data and over time; and   rendering, within the graphical user interface, a characterization of the determined performance metric for each of the subgroups, wherein the characterization of the determined performance metric indicates a relative proportion size of a respective subgroup of the live data.   
     
     
         12 . The method of  claim 1 , wherein the first performance metric includes rate of false positive, count of false positive, cost of false positive, cost of overestimate, cost of underestimate, benefit missed by false positive, true positive, benefit of true positive, benefit of minimizing false positive, benefit of maximizing true positive, or a combination thereof;
 wherein the second performance metric includes rate of false negative, count of false negative, cost of false negative, benefit missed by false negative, true negative, benefit of true negative, benefit of minimizing false negative, benefit of maximizing true negative, or a combination thereof.   
     
     
         13 . The method of  claim 1 , further comprising:
 monitoring a third generated model, the monitoring including determining a third performance value; and   rendering, within the graphical user interface and the plot, a third graphical object at a third location characterizing the third performance value.   
     
     
         14 . A method comprising:
 determining a plurality of candidate models using a model generator and a dataset, each of the plurality of candidate models including a respective model type;   determining a performance of each of the plurality of candidate models;   adjusting, based on the determined performance of each of the plurality of candidate models, a ratio of model types being generated; and   determining additional candidate models using a model generator and the dataset, the additional candidate models including respective model types, the determining additional candidate models according to the adjusted ratio of model types being generated.   
     
     
         15 . The method of  claim 14 , wherein each respective model type is one of a set of model types. 
     
     
         16 . The method of  claim 14 , further comprising:
 receiving data characterizing an objective;   wherein the adjusting is further based on the received objective.   
     
     
         17 . The method of  claim 14 , further comprising:
 identifying, based on the determined performance, one or more models from the plurality of candidate models; and   displaying, in a graphical user interface, data characterizing the determined performance of the identified one or more models.   
     
     
         18 . The method of  claim 14 , wherein the determining the plurality of candidate models is according to an initial ratio determined from historic performance of similar data sets. 
     
     
         19 . A system comprising:
 at least one data processor; and   memory storing instructions, which, when executed by the at least one data processor, cause the at least one data processor to perform operations comprising:   monitoring performance of a first generated model while the first generated model is deployed for use on live data, the monitoring including determining a first performance value of the first generated model;   monitoring performance of a second generated model while the second generated model is deployed for use on live data, the monitoring including determining a second performance value of the second generated model;   rendering, within a graphical user interface, a plot including a first axis and a second axis, the first axis including a characterization of a first performance metric and the second axis including a characterization of a second performance metric; and   rendering, within the graphical user interface and the plot, a first graphical object at a first location characterizing the first performance value and a second graphical object at a second location characterizing the second performance value.   
     
     
         20 . The system of  claim 19 , the operations further comprising:
 identifying, within the live data, subgroups of the live data;   determining a performance metric for the first generated model and for each of the subgroups of the live data and over time; and   rendering, within the graphical user interface, a characterization of the determined performance metric for each of the subgroups, wherein the characterization of the determined performance metric indicates a relative proportion size of a respective subgroup of the live data.