Method and apparatus for reward-based learning of improved systems management policies

Abstract

In one embodiment, the present invention is a method for reward-based learning of improved systems management policies. One embodiment of the inventive method involves supplying a first policy and a reward mechanism. The first policy maps states of at least one component of a data processing system to selected management actions, while the reward mechanism generates numerical measures of value responsive to particular actions (e.g., management actions) performed in particular states of the component(s). The first policy and the reward mechanism are applied to the component(s), and results achieved through this application (e.g., observations of corresponding states, actions and rewards) are processed in accordance with reward-based learning to derive a second policy having improved performance relative to the first policy in at least one state of the component(s).

1 . A method for learning a policy for management of at least one component of a data processing system, the method comprising: 
 obtaining a decision-making entity for managing said at least one component;    obtaining a reward mechanism for generating numerical measures of value responsive to at least one action performed in at least one state of said at least one component;    applying said decision-making entity and said reward mechanism to said at least one component;    processing a result achieved through application of said decision-making entity and said reward mechanism in accordance with reward-based learning; and    deriving said policy in accordance with said reward-based learning processing.    
   
   
       2 . The method of  claim 1 , wherein a performance measure associated with application of said reward mechanism and said policy to said at least one component is greater than a performance measure associated with application of said reward mechanism and said decision-making entity to said at least one component in at least one state of said at least one component.  
   
   
       3 . The method of  claim 1 , further comprising: 
 applying said reward mechanism and said policy to said at least one component.    
   
   
       4 . The method of  claim 3 , further comprising: 
 processing a result achieved through application of said reward mechanism and said policy in accordance with said reward-based learning; and    deriving a new policy in accordance with said reward-based learning processing.    
   
   
       5 . The method of  claim 1 , wherein said decision-making entity comprises at least one of: a human administrator, a rule-based method or a system model-based method.  
   
   
       6 . The method of  claim 1 , wherein said result comprises at least one observed state of said at least one component, at least one observed action responsive to said decision-making entity and at least one observed reward generated by said reward mechanism.  
   
   
       7 . The method of  claim 6 , wherein said result further comprises at least one observed transition of said at least one component from an initial state to a subsequent state.  
   
   
       8 . The method of  claim 7 , wherein said result further comprises at least one observed result of an internal calculation performed by said decision-making entity.  
   
   
       9 . The method of  claim 7 , wherein said result of said internal calculation is one or more expected-value estimates.  
   
   
       10 . The method of  claim 6 , wherein said result further comprises at least one exploratory off-policy action that differs from actions responsive to said decision-making entity.  
   
   
       11 . The method of  claim 1 , wherein said processing comprises: 
 learning a state transition model;    learning an expected-reward model; and    deriving said policy in accordance with said state transition model and said expected-reward model.    
   
   
       12 . The method of  claim 1 , wherein said reward-based learning comprises reinforcement learning.  
   
   
       13 . The method of  claim 12 , wherein said reinforcement learning comprises at least one of: value-function learning, actor-critic learning or direct policy learning.  
   
   
       14 . The method of  claim 1 , wherein said processing is performed off-line.  
   
   
       15 . The method of  claim 1 , wherein said processing is performed on-line.  
   
   
       16 . The method of  claim 1 , wherein said policy is applied to the allocation of one or more computing resources available to said at least one component, said one or more computing resources comprising at least one of: a physical computing resource, a virtual computing resource or power to a physical computing device.  
   
   
       17 . A computer readable medium containing an executable program for learning a policy for management of at least one component of a data processing system, where the program performs the steps of: 
 obtaining a decision-making entity for managing said at least one component;    obtaining a reward mechanism for generating numerical measures of value responsive to at least one action performed in at least one state of said at least one component;    applying said decision-making entity and said reward mechanism to said at least one component;    processing a result achieved through application of said decision-making entity and said reward mechanism in accordance with reward-based learning; and    deriving said policy in accordance with said reward-based learning processing.    
   
   
       18 . The computer readable medium of  claim 17 , wherein a performance measure associated with application of said reward mechanism and said policy to said at least one component is greater than a performance measure associated with application of said reward mechanism and said decision-making entity to said at least one component in at least one state of said at least one component.  
   
   
       19 . The computer readable medium of  claim 17 , wherein said reward-based learning comprises reinforcement learning.  
   
   
       20 . Apparatus for learning a policy for management of at least one component of a data processing system, the apparatus comprising: 
 means for obtaining a decision-making entity for managing said at least one component;    means for obtaining a reward mechanism for generating numerical measures of value responsive to at least one action performed in at least one state of said at least one component;    means for applying said decision-making entity and said reward mechanism to said at least one component;    means for processing a result achieved through application of said decision-making entity and said reward mechanism in accordance with reward-based learning; and    means for deriving said policy in accordance with said reward-based learning processing.