Instance weighted learning machine learning model

Abstract

An instance weighted learning (IWL) machine learning model. In one example embodiment, a method of employing an IWL machine learning model may include identifying a temporal sequence of reinforcement learning machine learning training instances with each of the training instances including a state-action pair, determining a first quality value for a first training instance in the temporal sequence of reinforcement learning machine learning training instances determining a second quality value for a second training instance in the temporal sequence of reinforcement learning machine learning training instances, associating the first quality value with the first training instance, and associating the second quality value with the second training instance. In this example embodiment, the first quality value is higher than the second quality value.

What is claimed is: 
     
         1 . A method of employing an instance weighted learning (IWL) machine learning model, the method comprising:
 identifying a temporal sequence of reinforcement learning machine learning training instances, each of the training instances including a state-action pair;   determining a first quality value for a first training instance in the temporal sequence of reinforcement learning machine learning training instances;   determining a second quality value for a second training instance in the temporal sequence of reinforcement learning machine learning training instances, the first quality value being higher than the second quality value;   associating the first quality value with the first training instance; and   associating the second quality value with the second training instance.   
     
     
         2 . The method as recited in  claim 1 , further comprising:
 training a classifier using the first training instance weighted with a first weighting factor that is a function of the first quality value, and   training the classifier using the second training instance weighted with a second weighting factor that is a function of the second quality value.   
     
     
         3 . The method as recited in  claim 2 , wherein the training of the classifier is influenced more by the first training instance than by the second training instance due to the first quality value being higher than the second quality value. 
     
     
         4 . The method as recited in  claim 2 , wherein:
 the first quality value is a represented by positive number which causes the first weighting factor to be a positive weighting factor;   the training the classifier using the first training instance weighted with the positive first weighting factor tends to encourage learning to support training instances that are similar to the first training instance;   the second quality value is a represented by negative number which causes the second weighting factor to be a negative weighting factor; and   the training the classifier using the second training instance weighted with the negative second weighting factor tends to discourage learning to support training instances that are similar to the second training instance.   
     
     
         5 . The method as recited in  claim 2 , wherein the classifier comprises a multilayer perceptron (MLP) neural network, another multilayer neural network, a decision tree, or a support vector machine. 
     
     
         6 . The method as recited in  claim 1 , wherein the determining the first quality value for the first training instance and the determining the second quality value for the second training instance include:
 determining a reward of a current training instance in the temporal sequence;   determining a first discounted portion of the reward for inclusion in the first quality value; and   determining a second discounted portion of the reward for inclusion in the second quality value,   wherein the first training instance and the second training instance occur previous to the current training instance in the temporal sequence.   
     
     
         7 . A non-transitory computer-readable medium storing a program configured to cause a processor to execute the method as recited in  claim 1 . 
     
     
         8 . A method of employing an instance weighted learning (IWL) machine learning model to train a classifier, the method comprising:
 identifying a set of machine learning training instances;   determining a first quality value for a first training instance in the set of machine learning training instances;   determining a second quality value for a second training instance in the set of machine learning training instances, the first quality value being higher than the second quality value;   associating the first quality value with the first training instance;   associating the second quality value with the second training instance;   training a classifier using the first training instance weighted using the first quality value; and   training the classifier using the second training instance weighted using the second quality value,   wherein the training of the classifier is influenced more by the first training instance than by the second training instance due to the first quality value being higher than the second quality value.   
     
     
         9 . The method as recited in  claim 8 , wherein:
 the weighting of the first training instance using the first quality value includes weighting the first training instance with a first weighting factor that is a function of the first quality value; and   the weighting of the second training instance using the second quality value includes weighting the second training instance with a second weighting factor that is a function of the second quality value.   
     
     
         10 . The method as recited in  claim 9 , wherein the set of machine learning training instances includes a temporal sequence of machine learning training instances. 
     
     
         11 . The method as recited in  claim 10 , wherein the determining the first quality value for the first training instance and the determining the second quality value for the second training instance include:
 determining a reward of a current training instance in the temporal sequence;   determining a first discounted portion of the reward for inclusion in the first quality value; and   determining a second discounted portion of the reward for inclusion in the second quality value.   
     
     
         12 . The method as recited in  claim 11 , wherein:
 the first training instance and the second training instance both occur previous to the current training instance in the temporal sequence; and   each of the first discounted portion and the second discounted portion is reduced the farther that corresponding training instance is positioned in the temporal sequence from the current training instance.   
     
     
         13 . The method as recited in  claim 8 , wherein:
 the first quality value is a represented by positive number which causes the first weighting factor to be a positive weighting factor;   the training the classifier using the first training instance weighted with the positive first weighting factor tends to encourage learning to support training instances that are similar to the first training instance;   the second quality value is a represented by negative number which causes the second weighting factor to be a negative weighting factor; and   the training the classifier using the second training instance weighted with the negative second weighting factor tends to discourage learning to support training instances that are similar to the second training instance.   
     
     
         14 . A non-transitory computer-readable medium storing a program configured to cause a processor to execute the method as recited in  claim 8 . 
     
     
         15 . A method of employing an instance weighted learning (IWL) machine learning model to train a classifier, the method comprising:
 identifying a temporal sequence of machine learning training instances;   determining a first quality value for a first training instance in the temporal sequence of machine learning training instances;   determining a second quality value for a second training instance in the temporal sequence of machine learning training instances, the first quality value being higher than the second quality value;   associating the first quality value with the first training instance;   associating the second quality value with the second training instance;   training a classifier using the first training instance weighted with a first weighting factor that is a function of the first quality value;   training the classifier using the second training instance weighted with a second weighting factor that is a function of the second quality value;   wherein the training of the classifier is influenced more by the first training instance than by the second training instance due to the first quality value being higher than the second quality value.   
     
     
         16 . The method as recited in  claim 15 , wherein the first weighting factor and the second weighting factor are determined according to any monotonically increasing function that satisfies the following formula:
   if ( q   1   ≧q   2 ), then ( u ( q   1 )≧ u ( q   2 )), where:
   q 1  is the first quality value;   q 2  is the second quality value;   u(q 1 ) is the first weighting factor; and   u(q 2 ) is the second weighting factor.   
     
     
         17 . The method as recited in  claim 15 , wherein:
 the first weighting factor is determined according the following formula:
     u ( q   1 )=( a+b·q   1 ); and 
   the second weighting factor is determined according the following formula:
     u ( q   2 )=( a+b·q   2 ), where 
   q 1  is the first quality value;   u(q 1 ) is the first weighting factor;   a is a first empirical parameter;   b is a second empirical parameter.   q 2  is the second quality value; and   u(q 2 ) is the second weighting factor.   
     
     
         18 . The method as recited in  claim 15 , wherein:
 each of the training instances is a multiple output dependency (MOD) machine learning training instance and each of the training instances includes multiple interdependent output components; and   the training the classifier using the first training instance and the training the classifier using the second training instance include employing a hierarchical based sequencing (HBS) machine learning model or a multiple output relaxation (MOR) machine learning model to train a separate classifier for each one of the multiple interdependent output components.   
     
     
         19 . The method as recited in  claim 18 , wherein each MOD machine learning training instance is a lead response management (LRM) MOD machine learning training instance. 
     
     
         20 . A non-transitory computer-readable medium storing a program configured to cause a processor to execute the method as recited in  claim 15 .