Predictive model validation

Abstract

Models are generated using a variety of tools and features of a model generation platform. For example, in connection with a project in which a user generates a predictive model based on historical data about a system being modeled, the user is provided through a graphical user interface a structured sequence of model generation activities to be followed, the sequence including dimension reduction, model generation, model process validation, and model re-generation. In connection with a project in which a user generates a predictive model based on historical data about a system being modeled, the user is enabled to validate the model development process with cross-validation between at least two subsets of the historical data; the validated model development process is enabled to be reapplied.

1 . A medium storing instructions that are executable by a processor to:
 in connection with a project,   select a first subset of historical data about a system being modeled for use in generating a tentative predictive model for the system, the first subset of historical data comprising less than all of the historical data;   generate the tentative predictive model by applying a selected model development process to the first subset of historical data;   determine a validity of the selected model development process based on the performance measures of the tentative predictive model; and   if the selected model development process is so validated, then apply the validated model development process to a full set of historical data that includes the first subset to generate a final predictive model.   
     
     
         2 . The medium of  claim 1  in which the validity of the selected model development process is determined by
 selecting a second subset of the historical data, the second subset being less than all of the historical data and including at least some historical data that is not in the first subset, or the second subset being randomly selected from the historical data and independent of the first subset; and 
 applying the tentative predictive model to the second subset of historical date. 
 
     
     
         3 . The medium of  claim 1  in which the selected model development process is from among multiple possible model development processes. 
     
     
         4 . The medium of  claim 1  in which the instructions are executable by a processor to:
 display goals of the project on a user interface to enable a user to assess performance of the project, wherein the project goals comprise at least one of: cumulative lift over an interval of interest, degree of monotonicity, or concordance scores. 
 
     
     
         5 . The medium of  claim 1  in which the instructions are executable by a processor to:
 enable a user to choose interactively at least one model development criterion change or transformation or interaction of variables to improve a fit of the tentative or final predictive model. 
 
     
     
         6 . The medium of  claim 1  in which the instructions are executable by a processor to:
 graphically display and compare the performance measures for a validation dataset and a training dataset selected from the historical data. 
 
     
     
         7 . The medium of  claim 1  in which applying the selected model development process includes (a) automatically transforming variables of the first subset of the historical data, (b) automatically generating the tentative predictive model, and (c) automatically generating performance measures of the tentative predictive model. 
     
     
         8 . The medium of  claim 1  in which the instructions are executable by a processor to:
 persistently store the validated model development process 
 
     
     
         9 . The medium of  claim 1  in which the instructions are executable by a processor to:
 enable a user to choose interactively at least one transformation or interaction of variables to improve the selected model development process. 
 
     
     
         10 . The medium of  claim 2  in which the instructions are executable by a processor to:
 enable a user to select the first subset and the second subset of historical data through a user interface. 
 
     
     
         11 . The medium of  claim 1  in which the instructions are executable by a processor to:
 enable a user to point and click to cause display of information about the validation of the model development process through a user interface. 
 
     
     
         12 . The medium of  claim 1  in which the instructions are executable by a processor to:
 enable a user to point and click to cause display of information about the performance of the validated model development process applied to the entire set of historical data through a user interface. 
 
     
     
         13 . The medium of  claim 1  in which the instructions are executable by a processor to:
 enable the user to select at least one machine automated model development process applied to the entire set of the historical data for the validated model development process through a user interface. 
 
     
     
         14 . A computer system comprising:
 a processor;   a memory; and   a storage device that stores a program for execution by the processor using the memory, the program comprising instructions for causing the processor to   in connection with a project,   select a first subset of historical data about a system being modeled for use in generating a tentative predictive model for the system, the first subset of historical data comprising less than all of the historical data;   generate the tentative predictive model by applying a selected model development process to the first subset of historical data;   determine a validity of the selected model development process based on performance measures of the tentative predictive model; and   if the selected model development process is so validated, then apply the validated model development process to a full set of historical data that includes the first subset to generate a final predictive model.   
     
     
         15 . The computer system of  claim 14  in which the validity of the selected model development process is determined by
 selecting a second subset of the historical data, the second subset being less than all of the historical data and including at least some historical data that is not in the first subset, or the second subset being randomly selected from the historical data and independent of the first subset; and 
 applying the tentative predictive model to the second subset of historical date. 
 
     
     
         16 . The computer system of  claim 14  in which the program comprises instructions for causing the processor to:
 display goals of the project on a user interface to enable a user to assess performance of the project, wherein the project goals comprise at least one of: 
 cumulative lift over an interval of interest, degree of monotonicity, or concordance scores. 
 
     
     
         17 . The computer system of  claim 14  in which the program comprises instructions for causing the processor to:
 enable a user to choose interactively at least one model development criterion change or transformation or interaction of variables to improve a fit of the tentative or final predictive model. 
 
     
     
         18 . The computer system of  claim 14  in which the program comprising instructions for causing the processor to:
 graphically display and compare the performance measures for a validation dataset and a training dataset selected from the historical data. 
 
     
     
         19 . The computer system of  claim 14  in which applying the selected model development process including (a) automatically transforming variables of the first subset of the historical data, (b) automatically generating the tentative predictive model, and (c) automatically generating performance measures of the tentative predictive model. 
     
     
         20 . The computer system of  claim 14  in which the program comprises instructions for causing the processor to:
 persistently store the validated model development process