Distributed resource-aware training of machine learning pipelines

Abstract

A method, a structure, and a computer system for predicting pipeline training requirements. The exemplary embodiments may include receiving one or more worker node features from one or more worker nodes, extracting one or more pipeline features from one or more pipelines to be trained, and extracting one or more dataset features from one or more datasets used to train the one or more pipelines. The exemplary embodiments may further include predicting an amount of one or more resources required for each of the one or more worker nodes to train the one or more pipelines using the one or more datasets based on one or more models that correlate the one or more worker node features, one or more pipeline features, and one or more dataset features with the one or more resources. Lastly, the exemplary embodiments may include identifying a worker node requiring a least amount of the one or more resources of the one or more worker nodes for training the one or more pipelines.

What is claimed is: 
     
       1. A computer-implemented method for predicting pipeline training requirements, the method comprising:
 receiving one or more worker node features from one or more worker nodes; 
 extracting one or more pipeline features from one or more pipelines to be trained; 
 extracting one or more dataset features from one or more datasets used to train the one or more pipelines; 
 predicting an amount of one or more resources required for each of the one or more worker nodes to train the one or more pipelines using the one or more datasets based on one or more models that correlate the one or more worker node features, one or more pipeline features, and one or more dataset features with the one or more resources, the amount of the one or more resources including a peak memory utilization; 
 identifying a worker node requiring a least amount of the one or more resources of the one or more worker nodes for training at least one pipeline of the one or more pipelines; and 
 training the at least one pipeline using the worker node. 
 
     
     
       2. The method of  claim 1 , further comprising:
 determining an actual amount of resources required by the worker node to train the one or more pipelines; and 
 adjusting the one or more models based on comparing the predicted amount of resources to the actual amount of resources. 
 
     
     
       3. The method of  claim 1 , wherein the one or more worker node features respectively include a number of CPUs and cores therein, a number of GPUs and cores therein, CPU utilization, GPU utilization, CPU memory, GPU memory, CPU and GPU swap usage, and outputs of vmstat and iostat commands. 
     
     
       4. The method of  claim 1 , wherein the one or more pipeline features include a type of estimator, a type of pre-processor, a type of feature engineering, and parameter settings thereof. 
     
     
       5. The method of  claim 1 , wherein the one or more dataset features include a number of datapoints, a number of features, a number of cross-validation folds, a number of features with categorical values, a number of features with real values, a number of missing values, and a number of sparse values. 
     
     
       6. The method of  claim 1 , wherein the one or more models are trained via Random Forest, GBM, Logistic Regression, Deep Neural Networks, and Autoencoders. 
     
     
       7. The method of  claim 1 , wherein the one or more resources include a training time, a power consumption, and a peak CPU utilization. 
     
     
       8. A computer program product for predicting pipeline training requirements, the computer program product comprising:
 one or more non-transitory computer-readable storage media and program instructions stored on the one or more non-transitory computer-readable storage media capable of performing a method, the method comprising:
 receiving one or more worker node features from one or more worker nodes; 
 extracting one or more pipeline features from one or more pipelines to be trained; 
 extracting one or more dataset features from one or more datasets used to train the one or more pipelines; 
 predicting an amount of one or more resources required for each of the one or more worker nodes to train the one or more pipelines using the one or more datasets based on one or more models that correlate the one or more worker node features, one or more pipeline features, and one or more dataset features with the one or more resources, the amount of the one or more resources including a peak memory utilization; 
 identifying a worker node requiring a least amount of the one or more resources of the one or more worker nodes for training at least one pipeline of the one or more pipelines; and 
 training the at least one pipeline using the worker node. 
 
 
     
     
       9. The computer program product of  claim 8 , further comprising:
 determining an actual amount of resources required by the worker node to train the one or more pipelines; and 
 adjusting the one or more models based on comparing the predicted amount of resources to the actual amount of resources. 
 
     
     
       10. The computer program product of  claim 8 , wherein the one or more worker node features respectively include a number of CPUs and cores therein, a number of GPUs and cores therein, CPU utilization, GPU utilization, CPU memory, GPU memory, CPU and GPU swap usage, and outputs of vmstat and iostat commands. 
     
     
       11. The computer program product of  claim 8 , wherein the one or more pipeline features include a type of estimator, a type of pre-processor, a type of feature engineering, and parameter settings thereof. 
     
     
       12. The computer program product of  claim 8 , wherein the one or more dataset features include a number of datapoints, a number of features, a number of cross-validation folds, a number of features with categorical values, a number of features with real values, a number of missing values, and a number of sparse values. 
     
     
       13. The computer program product of  claim 8 , wherein the one or more models are trained via Random Forest, GBM, Logistic Regression, Deep Neural Networks, and Autoencoders. 
     
     
       14. The computer program product of  claim 8 , wherein the one or more resources include a training time, a power consumption, and a peak CPU utilization. 
     
     
       15. A computer system for predicting pipeline training requirements, the system comprising:
 one or more computer processors, one or more computer-readable storage media, and program instructions stored on the one or more of the computer-readable storage media for execution by at least one of the one or more processors capable of performing a method, the method comprising:
 receiving one or more worker node features from one or more worker nodes; 
 extracting one or more pipeline features from one or more pipelines to be trained; 
 extracting one or more pipeline features from one or more pipelines to be trained; 
 predicting an amount of one or more resources required for each of the one or more worker nodes to train the one or more pipelines using the one or more datasets based on one or more models that correlate the one or more worker node features, one or more pipeline features, and one or more dataset features with the one or more resources, the amount of the one or more resources including a peak memory utilization; 
 identifying a worker node requiring a least amount of the one or more resources of the one or more worker nodes for training at least one pipeline of the one or more pipelines; and 
 training the at least one pipeline using the worker node. 
 
 
     
     
       16. The computer system of  claim 15 , further comprising:
 determining an actual amount of resources required by the worker node to train the one or more pipelines; and 
 adjusting the one or more models based on comparing the predicted amount of resources to the actual amount of resources. 
 
     
     
       17. The computer system of  claim 15 , wherein the one or more worker node features respectively include a number of CPUs and cores therein, a number of GPUs and cores therein, CPU utilization, GPU utilization, CPU memory, GPU memory, CPU and GPU swap usage, and outputs of vmstat and iostat commands. 
     
     
       18. The computer system of  claim 15 , wherein the one or more pipeline features include a type of estimator, a type of pre-processor, a type of feature engineering, and parameter settings thereof. 
     
     
       19. The computer system of  claim 15 , wherein the one or more dataset features include a number of datapoints, a number of features, a number of cross-validation folds, a number of features with categorical values, a number of features with real values, a number of missing values, and a number of sparse values. 
     
     
       20. The computer system of  claim 15 , wherein the one or more models are trained via Random Forest, GBM, Logistic Regression, Deep Neural Networks, and Autoencoders.