Selection of machine learning algorithms

Abstract

Systems and methods of selecting machine learning models/algorithms for a candidate dataset are disclosed. A computer system may access historical data of a set of algorithms applied to a set of benchmark datasets; select a first algorithm of the set of algorithms; apply the first algorithm to an input dataset to create a model of the input dataset; evaluate and store results of the applying; and add the first algorithm to a set of tried algorithms. The computer system may select a next algorithm of the algorithm set via submodular optimization based on the historical data and the set of tried algorithms; apply the next algorithm to the input dataset; capture a next result based on the applying; add the next result to update the set of tried algorithms; and repeat the submodular optimization. The procedure may continue until a termination condition is reached.

1 . A computer-readable medium including instructions, which when executed by one or more processors of a computing system, causes the computing system to:
 identify a set of algorithms;   identify a set of benchmark datasets;   generate a set of predictions by application of individual algorithms of the set of algorithms to individual benchmark datasets of the set of benchmark datasets;   evaluate the set of predictions to obtain results;   control storage of the results in a benchmark database;   generate a submodular function based on the stored results;   apply the individual algorithms to a candidate dataset;   identify an optimum algorithm, wherein the optimum algorithm is an individual algorithm of the set of algorithms that is closest to fulfilling a predetermined criterion than other algorithms; and   control transmission of a report indicating the optimum algorithm.   
     
     
         2 . The computer-readable medium of  claim 1 , wherein, in response to execution of the instructions, is to:
 add the individual algorithms to a list of algorithms after application the individual algorithms to the candidate dataset, and   wherein, to identify the optimum algorithm, the computing system, in response to execution of the instructions, is to identify the optimum algorithm from among the individual algorithms in the list of algorithms based on the predetermined criterion.   
     
     
         3 . The computer-readable medium of  claim 2 , wherein, in response to execution of the instructions, is to:
 evaluate another set of predictions to obtain other results, wherein the other set of predictions is obtained from the application of the individual algorithms to the candidate dataset; and   control storage of the other results in the benchmark database, and   wherein, to identify the optimum algorithm from among the individual algorithms, the computing system, in response to execution of the instructions, is to identify the optimum algorithm from among the individual algorithms in the list of algorithms further based on the stored other results.   
     
     
         4 . The computer-readable medium of  claim 1 , wherein:
 to identify the set of datasets, the computing system, in response to execution of the instructions, is to obtain the set of datasets from a benchmark datasets database, and   to identify the set of algorithms, the computing system, in response to execution of the instructions, is to obtain the set of algorithms from a modeling algorithms database.   
     
     
         5 . The computer-readable medium of  claim 1 , wherein, to generate the set of predictions, the computing system, in response to execution of the instructions, is to:
 control execution of the individual algorithms using, as an input, data of the individual benchmark datasets, and   wherein the set of predictions comprises an output of the execution of the individual algorithms with the data of the individual datasets as the input.   
     
     
         6 . The computer-readable medium of  claim 1 , wherein, to evaluate the set of predictions, the computing system, in response to execution of the instructions, is to:
 control performance of a holdout procedure using the set of predictions; or   control performance of a two-fold cross-validation procedure using the set of predictions.   
     
     
         7 . The computer-readable medium of  claim 1 , wherein to control storage of the results, the computing system, in response to execution of the instructions, is to:
 control storage, in the benchmark database, of identifiers of the individual algorithms and identifiers of the benchmark datasets in association with the individual algorithms.   
     
     
         8 . The computer-readable medium of  claim 1 , wherein, to apply the individual algorithms to the candidate dataset, the computing system, in response to execution of the instructions, is to:
 control execution of the individual algorithms using, as an input, data of the candidate dataset; and   control storage of an output of the execution of the individual algorithms with the data of the candidate dataset as the input.   
     
     
         9 . A computing system comprising:
 one or more processors to implement a prediction engine, the prediction engine to:
 obtain a candidate dataset from a user database; 
 identify a first algorithm from a set of algorithms; 
 apply the first algorithm to the candidate dataset to obtain first predictions; 
 evaluate the first predictions to obtain first results; 
 store the first results in a database; 
 add the first algorithm to an algorithms list; 
 generate a submodular function based on the stored benchmark data; 
 for each algorithm in the set of algorithms,
 select a next algorithm from the set of algorithms based on the submodular function, 
 apply the next algorithm to the candidate dataset to obtain next predictions, 
 evaluate the next predictions to obtain next results, and 
 add the next algorithm to the algorithms list; 
 
 identify an optimum algorithm from the algorithms list, wherein the optimum algorithm is an individual algorithm of the algorithms list that is closer to fulfilling a predetermined criterion than other algorithms of the algorithms list; and 
 generate a report indicating the optimum algorithm; and 
   a network interface to transmit the report to a user system.   
     
     
         10 . The computing system of  claim 9 , further comprising:
 a one or more processors to implement a modeling engine to:
 obtain another set of algorithms from a modeling algorithms database, 
 obtain a set of benchmark datasets from the benchmark dataset database, 
 apply individual algorithms of the set of algorithms to individual benchmark datasets of the set of benchmark datasets to obtain benchmark predictions for the individual algorithms, 
 evaluate the benchmark predictions to obtain benchmark results for the individual algorithms, and 
 store, as the benchmark data, the benchmark results in a benchmark database, 
   wherein the prediction engine is to obtain the set of algorithms from the benchmark database based on the benchmark results.   
     
     
         11 . The computing system of  claim 10 , wherein, to apply the individual algorithms to the individual benchmark datasets, the modeling engine is to:
 execute the individual algorithms using data of the individual benchmark datasets as an input to the individual algorithms, and wherein the benchmark predictions comprise an output of the execution of the individual algorithms.   
     
     
         12 . The computing system of  claim 10 , wherein, to evaluate the benchmark predictions, the modeling engine is to:
 perform a holdout procedure using the benchmark predictions; or   perform a two-fold cross-validation procedure using the benchmark predictions.   
     
     
         13 . The computing system of  claim 10 , wherein, to store the benchmark results, the modeling engine is to:
 store, in the benchmark database, identifiers of the individual algorithms and identifiers of the benchmark datasets in association with the individual algorithms and the benchmark results.   
     
     
         14 . The computing system of  claim 9 , wherein, to apply the first algorithm to the candidate dataset, the prediction engine is to:
 execute the first algorithm using candidate data of the candidate dataset as an input to the first algorithm, and wherein the first predictions comprise an output of the execution of the first algorithm.   
     
     
         15 . The computing system of  claim 9 , wherein, to apply the next algorithm to the candidate dataset, the prediction engine is to:
 execute the next algorithm using candidate data of the candidate dataset as an input to the next algorithm, and wherein the next predictions comprise an output of the execution of the next algorithm.   
     
     
         16 . The computing system of  claim 9 , wherein, to evaluate the first predictions or to evaluate the next predictions, the prediction engine is to:
 perform a holdout procedure using the first predictions or the next predictions; or   perform a two-fold cross-validation procedure using the first predictions or the next predictions.   
     
     
         17 . The computing system of  claim 10 , further comprising:
 one or more processors to implement a source engine to obtain, from the user system, raw data or an indication of a location from which the raw data is to be obtained, convert the raw data into a user dataset, and store the user dataset in the user database; and   one or more processors to implement a modeling server to obtain a set of algorithms from a modeling algorithms database, obtain a set of benchmark datasets from the benchmark dataset database, apply individual algorithms of the set of algorithms to individual benchmark datasets of the set of benchmark datasets to obtain benchmark predictions for the individual algorithms, evaluate the benchmark predictions to obtain benchmark results for the individual algorithms, and store the benchmark results in a benchmark database.   
     
     
         18 . A computing system comprising:
 a prediction server to:
 obtain an input dataset; 
 select a first algorithm from a set of algorithms; 
 apply the first algorithm to the input dataset to create a first model of the input dataset; 
 evaluate and control storage of results of applying the first algorithm to the input dataset; 
 add the first algorithm into a set of tried algorithms; 
 for each algorithm in the set of algorithms other than the first algorithm and until a termination condition is met:
 select, via submodular optimization, a next algorithm of the set of algorithms to apply to the input dataset based accessed historical performance data in a benchmark database and the set of tried algorithms, 
 evaluate and control storage of results of applying the next algorithm to the input dataset, 
 add the next algorithm to the set of tried algorithms; 
 
 identify an optimum algorithm from the algorithms list, wherein the optimum algorithm is an individual algorithm of the algorithms list that is closer to fulfilling a predetermined criterion than other algorithms of the algorithms list; 
 generate a report indicating the optimum algorithm; and 
   an application server to implement a user interface, the application server to obtain the report, and transmit the report to the user system.   
     
     
         19 . The computing system of  claim 18 , further comprising:
 a modeling server to obtain a set of algorithms from a modeling algorithms database, obtain a set of benchmark datasets from the benchmark dataset database, apply individual algorithms of the set of algorithms to individual benchmark datasets of the set of benchmark datasets to obtain benchmark predictions for the individual algorithms, evaluate the benchmark predictions to obtain benchmark results for the individual algorithms, and store the benchmark results in a benchmark database.   
     
     
         20 . The computing system of  claim 19 , wherein the prediction server is further to generate a submodular function based on the stored first results and select the next algorithm using the submodular function.