Systems and methods for training machine learning models

Abstract

Methods and computer-readable media for repeated holdout validation include collecting independent data representing independent variables; collecting dependent data representing a dependent variable; correlating the independent data with the dependent data; creating a data set comprising the correlated independent and dependent data; generating a plurality of unique seeds; creating a plurality of training sets and a plurality of validation sets; associating each training set with a single validation set; training the neural network a plurality of times with the training sets and seeds to create a plurality of models; calculating accuracy metric values for the models using the validation sets associated with the training sets used to create respective models; performing a statistical analysis of the accuracy metric values; and ranking the independent variables by a strength of correlation of individual independent variables with the dependent variable, when a metric of the statistical analysis exceeds a threshold.

What is claimed is: 
     
       1. A method comprising:
 collecting independent data representing independent variables; 
 collecting dependent data representing a dependent variable; 
 correlating the independent data with the dependent data; 
 creating a data set comprising the correlated independent and dependent data; 
 generating a plurality of unique seeds; 
 creating a plurality of training sets and a plurality of validation sets; 
 associating each training set with a single validation set; 
 training a neural network a plurality of times with the training sets and seeds to create a plurality of models; 
 calculating accuracy metric values for the models using the validation sets associated with the training sets used to create respective models; 
 performing a statistical analysis of the accuracy metric values; and 
 ranking the independent variables by a strength of correlation of individual independent variables with the dependent variable, when a metric of the statistical analysis exceeds a threshold. 
 
     
     
       2. The method of  claim 1 , wherein:
 the dependent variable comprises a user satisfaction metric; and 
 the independent variables comprise business transaction variables. 
 
     
     
       3. The method of  claim 2 , wherein:
 the dependent data comprises data generated from surveys; and 
 the independent data comprises data generated by analysis of business records. 
 
     
     
       4. The method of  claim 3 , wherein:
 the dependent data comprises a net value calculated by analysis of survey results. 
 
     
     
       5. The method of  claim 3 , wherein the business records comprise at least one of:
 sales value; 
 sales quantity; 
 sales frequency; 
 transaction time; 
 user referrals; or 
 a success ratio, calculated as a ratio of completed transactions to initiated transactions. 
 
     
     
       6. The method of  claim 1 , wherein the unique seeds are generated by one of random or pseudo-random procedures. 
     
     
       7. The method of  claim 1 , wherein:
 the accuracy metric comprises an Area Under the Curve (AUC); 
 the statistical analysis comprises calculating an average; and 
 the metric of the statistical analysis comprises the average. 
 
     
     
       8. The method of  claim 1 , wherein associating each training set with a single validation set further comprises:
 pairing training and validation sets such that no individual data point is in both a training set and a validation set of a pair. 
 
     
     
       9. The method of  claim 8 , wherein:
 a number of training sets and a number of validation sets is equal to a number of unique seeds; 
 a number of the models is equal to the number of unique seeds; and 
 each model is created by training with one seed and one pair of a training set and a validation set. 
 
     
     
       10. The method of  claim 8 , wherein:
 individual data points are assigned to a training set or a validation set based on one of a random number or a pseudo-random number. 
 
     
     
       11. A non-transitory computer-readable medium containing instructions to perform operations comprising:
 collecting independent data representing independent variables; 
 collecting dependent data representing a dependent variable; 
 correlating the independent data with the dependent data; 
 creating a data set comprising the correlated independent and dependent data; 
 generating a plurality of unique seeds; 
 creating a plurality of training sets and a plurality of validation sets; 
 associating each training set with a single validation set; 
 training a neural network a plurality of times with the training sets and seeds to create a plurality of models; 
 calculating accuracy metric values for the models using the validation sets associated with the training sets used to create respective models; 
 performing a statistical analysis of the accuracy metric values; and 
 ranking the independent variables by a strength of correlation of individual independent variables with the dependent variable, when a metric of the statistical analysis exceeds a threshold. 
 
     
     
       12. The medium of  claim 11 , wherein:
 the dependent variable comprises a user satisfaction metric; and 
 the independent variables comprise business transaction variables. 
 
     
     
       13. The medium of  claim 12 , wherein:
 the dependent data comprises data generated from surveys; and 
 the independent data comprises data generated by analysis of business records. 
 
     
     
       14. The medium of  claim 13 , wherein:
 the dependent data comprises a net value calculated by analysis of survey results. 
 
     
     
       15. The medium of  claim 13 , wherein the business records comprise at least one of:
 sales value; 
 sales quantity; 
 sales frequency; 
 transaction time; 
 user referrals; or 
 a success ratio, calculated as a ratio of completed transactions to initiated transactions. 
 
     
     
       16. The medium of  claim 11 , wherein the unique seeds are generated by one of random or pseudo-random procedures. 
     
     
       17. The medium of  claim 11 , wherein:
 the accuracy metric comprises an Area Under the Curve (AUC); 
 the statistical analysis comprises calculating an average; and 
 the metric of the statistical analysis comprises the average. 
 
     
     
       18. The medium of  claim 11 , wherein associating each training set with a single validation set further comprises:
 pairing training and validation sets such that no individual data point is in both a training set and a validation set of a pair. 
 
     
     
       19. The medium of  claim 18 , wherein:
 a number of training sets and a number of validation sets is equal to a number of unique seeds; 
 a number of the models is equal to the number of unique seeds; and 
 each model is created by training with one seed and one pair of a training set and a validation set. 
 
     
     
       20. The medium of  claim 18 , wherein:
 individual data points are assigned to a training set or a validation set based on one of a random number or a pseudo-random number.