Deep reinforcement learning for workflow optimization using provenance-based simulation

Abstract

Deep reinforcement learning techniques and provenance-based simulation are employed for resource allocation in a shared computing environment. One method comprises: obtaining a specification of a workflow of concurrent workflows in a shared computing environment, wherein the specification comprises workflow states and one or more control variables for the workflow in the shared computing environment; obtaining a simulation model of the workflow representing different configurations of the control variables; evaluating the control variables for the concurrent workflows using a reinforcement learning (RL) agent by observing the states and obtaining an expected utility score for control variable combinations for the execution of the concurrent workflows given an allocation of resources of the shared computing environment, wherein the RL agent performs, using the simulation model, the evaluating, the obtaining the expected utility score, and/or a training of an RL model; and providing an allocation of the resources based on the expected utility score.

What is claimed is: 
     
       1. A method, comprising:
 obtaining a specification of at least one workflow of a plurality of concurrent workflows in a shared computing environment, wherein the specification comprises a plurality of states of the at least one workflow and one or more control variables indicating an allocation of one or more resources for the at least one workflow in the shared computing environment; 
 obtaining a simulation model that simulates the at least one workflow of the plurality of concurrent workflows representing a plurality of different configurations of the one or more control variables of the at least one workflow of the concurrent workflows by mapping the states of the at least one workflow based on a similarity given by one or more state similarity functions; 
 evaluating, using at least one processing device, a plurality of values of the one or more control variables for an execution of said plurality of concurrent workflows using at least one reinforcement learning agent, wherein said evaluating comprises observing said plurality of states, including a current state comprising a current configuration of said plurality of concurrent workflows and said shared computing environment, and obtaining an expected utility score for a plurality of combinations of said control variables for the execution of said plurality of concurrent workflows given an allocation of the one or more resources of the shared computing environment corresponding to said combination of said control variables in said current state, wherein the at least one reinforcement learning agent, using one or more training samples from one or more simulations of the at least one workflow by the simulation model, trains a reinforcement learning model used by the at least one reinforcement learning agent; and 
 providing an allocation of the one or more resources of the shared computing environment reflecting the combination of the control variables having the expected utility score that satisfies one or more predefined score criteria. 
 
     
     
       2. The method of  claim 1 , wherein the evaluating the plurality of values of the one or more control variables for the execution of said plurality of concurrent workflows using the at least one reinforcement learning agent further comprises observing the current state and selecting an action based on a path in the simulation model that substantially maximizes at least one utility function for one or more nodes in the simulation model. 
     
     
       3. The method of  claim 2 , wherein the action is selected based on the path in the simulation model when a configurable threshold satisfies one or more predefined value criteria. 
     
     
       4. The method of  claim 1 , wherein estimated values of the expected utility score are given by observing the current state and the estimated values of the expected utility score are estimated based on a path in the simulation model that substantially maximizes at least one utility function for one or more nodes in the simulation model for a predefined number of training epochs. 
     
     
       5. The method of  claim 1 , wherein the reinforcement learning model used by the at least one reinforcement learning agent is trained using the training samples, wherein the training samples comprise input/output training pairs generated from the simulation model as a training batch for a predefined number of training epochs. 
     
     
       6. The method of  claim 1 , wherein said expected utility score further comprises an expected cost depending on one or more of an execution time of the at least one workflow and a consumption of resources in said shared computing environment. 
     
     
       7. The method of  claim 1 , wherein said at least one reinforcement learning agent comprises a Deep Q-Learning agent using a Q-Deep Neural Network (QDNN) as a representation of a Q-Function, and wherein said obtaining the expected utility score for the plurality of combinations of said control variables comprises selecting an action at random and computing a cost-to-go from the expected utility score of the selected action updated by an observation of the current state, and wherein an updating of the at least one reinforcement learning agent comprises a training of the QDNN given new samples in iterative epochs. 
     
     
       8. The method of  claim 1 , wherein the one or more control variables comprise one or more of a number of processing cores allocated to a given workflow and an amount of memory allocated to the given workflow. 
     
     
       9. A system, comprising:
 a memory; and 
 at least one processing device, coupled to the memory, operative to implement the following steps: 
 obtaining a specification of at least one workflow of a plurality of concurrent workflows in a shared computing environment, wherein the specification comprises a plurality of states of the at least one workflow and one or more control variables indicating an allocation of one or more resources for the at least one workflow in the shared computing environment; 
 obtaining a simulation model that simulates the at least one workflow of the plurality of concurrent workflows representing a plurality of different configurations of the one or more control variables of the at least one workflow of the concurrent workflows by mapping the states of the at least one workflow based on a similarity given by one or more state similarity functions; 
 evaluating, using at least one processing device, a plurality of values of the one or more control variables for an execution of said plurality of concurrent workflows using at least one reinforcement learning agent, wherein said evaluating comprises observing said plurality of states, including a current state comprising a current configuration of said plurality of concurrent workflows and said shared computing environment, and obtaining an expected utility score for a plurality of combinations of said control variables for the execution of said plurality of concurrent workflows given an allocation of the one or more resources of the shared computing environment corresponding to said combination of said control variables in said current state, wherein the at least one reinforcement learning agent, using one or more training samples from one or more simulations of the at least one workflow by the simulation model, trains a reinforcement learning model used by the at least one reinforcement learning agent; and 
 providing an allocation of the one or more resources of the shared computing environment reflecting the combination of the control variables having the expected utility score that satisfies one or more predefined score criteria. 
 
     
     
       10. The system of  claim 9 , wherein the evaluating the plurality of values of the one or more control variables for the execution of said plurality of concurrent workflows using the at least one reinforcement learning agent further comprises observing the current state and selecting an action based on a path in the simulation model that substantially maximizes at least one utility function for one or more nodes in the simulation model. 
     
     
       11. The system of  claim 9 , wherein estimated values of the expected utility score are given by observing the current state and the estimated values of the expected utility score are estimated based on a path in the simulation model that substantially maximizes at least one utility function for one or more nodes in the simulation model for a predefined number of training epochs. 
     
     
       12. The system of  claim 9 , wherein the reinforcement learning model used by the at least one reinforcement learning agent is trained using the training samples, wherein the training samples comprise input/output training pairs generated from the simulation model as a training batch for a predefined number of training epochs. 
     
     
       13. The system of  claim 9 , wherein said expected utility score further comprises an expected cost depending on one or more of an execution time of the at least one workflow and a consumption of resources in said shared computing environment. 
     
     
       14. The system of  claim 9 , wherein said at least one reinforcement learning agent comprises a Deep Q-Learning agent using a Q-Deep Neural Network (QDNN) as a representation of a Q-Function, and wherein said obtaining the expected utility score for the plurality of combinations of said control variables comprises selecting an action at random and computing a cost-to-go from the expected utility score of the selected action updated by an observation of the current state, and wherein an updating of the at least one reinforcement learning agent comprises a training of the QDNN given new samples in iterative epochs. 
     
     
       15. A computer program product, comprising a non-transitory machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:
 obtaining a specification of at least one workflow of a plurality of concurrent workflows in a shared computing environment, wherein the specification comprises a plurality of states of the at least one workflow and one or more control variables indicating an allocation of one or more resources for the at least one workflow in the shared computing environment; 
 obtaining a simulation model that simulates the at least one workflow of the plurality of concurrent workflows representing a plurality of different configurations of the one or more control variables of the at least one workflow of the concurrent workflows by mapping the states of the at least one workflow based on a similarity given by one or more state similarity functions; 
 evaluating, using at least one processing device, a plurality of values of the one or more control variables for an execution of said plurality of concurrent workflows using at least one reinforcement learning agent, wherein said evaluating comprises observing said plurality of states, including a current state comprising a current configuration of said plurality of concurrent workflows and said shared computing environment, and obtaining an expected utility score for a plurality of combinations of said control variables for the execution of said plurality of concurrent workflows given an allocation of the one or more resources of the shared computing environment corresponding to said combination of said control variables in said current state, wherein the at least one reinforcement learning agent, using one or more training samples from one or more simulations of the at least one workflow by the simulation model, trains a reinforcement learning model used by the at least one reinforcement learning agent; and 
 providing an allocation of the one or more resources of the shared computing environment reflecting the combination of the control variables having the expected utility score that satisfies one or more predefined score criteria. 
 
     
     
       16. The computer program product of  claim 15 , wherein the evaluating the plurality of values of the one or more control variables for the execution of said plurality of concurrent workflows using the at least one reinforcement learning agent further comprises observing the current state and selecting an action based on a path in the simulation model that substantially maximizes at least one utility function for one or more nodes in the simulation model. 
     
     
       17. The computer program product of  claim 15 , wherein estimated values of the expected utility score are given by observing the current state and the estimated values of the expected utility score are estimated based on a path in the simulation model that substantially maximizes at least one utility function for one or more nodes in the simulation model for a predefined number of training epochs. 
     
     
       18. The computer program product of  claim 15 , wherein the reinforcement learning model used by the at least one reinforcement learning agent is trained using the training samples, wherein the training samples comprise input/output training pairs generated from the simulation model as a training batch for a predefined number of training epochs. 
     
     
       19. The computer program product of  claim 15 , wherein said expected utility score further comprises an expected cost depending on one or more of an execution time of the at least one workflow and a consumption of resources in said shared computing environment. 
     
     
       20. The computer program product of  claim 15 , wherein said at least one reinforcement learning agent comprises a Deep Q-Learning agent using a Q-Deep Neural Network (QDNN) as a representation of a Q-Function, and wherein said obtaining the expected utility score for the plurality of combinations of said control variables comprises selecting an action at random and computing a cost-to-go from the expected utility score of the selected action updated by an observation of the current state, and wherein an updating of the at least one reinforcement learning agent comprises a training of the QDNN given new samples in iterative epochs.