Methods and systems for feature selection in machine learning based on feature contribution and model fitness

Abstract

Methods and systems are provided for feature selection in machine learning, in which the features selected for inclusion in a prediction rule are selected based on statistical metric(s) of feature contribution and/or model fitness.

1 . A method for selecting features for a final prediction rule, said method comprising:
 (a) generating a prediction rule based on data for a set of features, wherein initially said set of features includes n features;   (b) determining a fitness value for said prediction rule;   (c) determining a value of contribution to said prediction rule for each of said features in said set of features;   (d) removing a feature from consideration from said set of features based on the values of contribution;   (e) iterating (a)-(d) in order to produce n prediction rules and n fitness values; and   (f) selecting, based on the fitness values for said n prediction rules, one of said n models as said final prediction rule.   
     
     
         2 . The method of  claim 1 , wherein stages (a)-(f) are performed in that order. 
     
     
         3 . The method of  claim 1 , wherein determining a fitness value for said prediction rule comprises summing a concordance index (CI) of said prediction rule with a product of a sensitivity and a specificity of said prediction rule. 
     
     
         4 . The method of  claim 1 , wherein determining a fitness value for said prediction rule comprises measuring a separation between one or more outcome value(s) predicted by said prediction rule and actual outcome(s). 
     
     
         5 . The method of  claim 1 , wherein determining a fitness value for said prediction rule comprises calculating the following risk functional:
     R ( k,l )=fitness/(1−sqrt(( k (log( l/k )+1)+log( l )/2)/ l ))   
       where l is a number of instances in said data for said set of n features, k is a VC-dimension of a set of functions from which said prediction rule is selected, and fitness is a function that evaluates quality of fit of said prediction rule. 
     
     
         6 . The method of  claim 1 , wherein said determining a value of contribution comprises determining a contribution value for each feature i by multiplying a weight of said feature i in said prediction rule with a measure of the discriminative ability of said feature i as observed in the data used to generate said prediction rule. 
     
     
         7 . The method of  claim 1 , wherein said determining a value of contribution comprises determining a contribution value for each feature i by multiplying a weight of said feature i in said prediction rule with a concordance index (CI) for said feature i. 
     
     
         8 . The method of  claim 1 , wherein said iterating (a)-(d) comprises:
 repeating (a) and (b) as long as said set of features includes 1 or more features; and   repeating (c) and (d) as long as said set of features includes 2 or more features.   
     
     
         9 . An apparatus for determining an outcome for an instance, said apparatus comprising:
 a computer implementation of a final prediction rule, wherein said final prediction rule is based on features selected through machine learning, said machine learning comprising (a) generating a prediction rule based on data for a set of features, wherein initially said set includes n features, (b) determining a fitness value for said prediction rule, (c) determining a value of contribution to said prediction rule for each of said features in said set of features, (d) removing a feature from consideration from said set of features based on the values of contribution, (e) iterating (a)-(d) in order to produce n prediction rules and n fitness values, and (f) selecting, based on the fitness values for said n models, one of said n prediction rules as said final prediction rule, wherein said computer implementation of said final prediction rule is configured to:   receive data for an instance; and   evaluate said data for said instance according to said final prediction rule, thereby determining an outcome for said instance.   
     
     
         10 . The apparatus of  claim 9 , wherein said machine learning determines a fitness value for said prediction rule by summing a concordance index (CI) of said prediction rule with a product of a sensitivity and a specificity of said prediction rule. 
     
     
         11 . The apparatus of  claim 9 , wherein said machine learning determines a fitness value for said prediction rule by measuring a separation between one or more outcome value(s) predicted by said prediction rule and actual outcome(s). 
     
     
         12 . The apparatus of  claim 9 , wherein said machine learning determines a fitness value for said prediction rule by calculating the following risk functional:
     R ( k,l )=fitness/(1−sqrt(( k (log( l/k )+1)+log( l )/2)/ l ))   
       where l is a number of instances in said data for said set of n features, k is a VC-dimension of a set of functions from which said prediction rule is selected, and fitness is a function that evaluates quality of fit of said prediction rule. 
     
     
         13 . The apparatus of  claim 9 , wherein said machine learning determines a value of contribution for each feature i by multiplying a weight of said feature i in said prediction rule with a measure of the discriminative ability of said feature i as observed in the data used to generate said prediction rule. 
     
     
         14 . The apparatus of  claim 9 , wherein said machine learning determines a value of contribution for each feature i by multiplying a weight of said feature i in said prediction rule with a concordance index (CI) for said feature i. 
     
     
         15 . A computer readable medium comprising computer executable instructions recorded thereon for performing the method comprising:
 (a) generating a prediction rule based on data for a set of features, wherein initially said set includes n features;   (b) determining a fitness value for said prediction rule;   (c) determining a value of contribution to said prediction rule for each of said features in said set of features;   (d) removing a feature from consideration from said set of features based on the values of contribution;   (e) iterating (a)-(d) to produce n prediction rules and n fitness values; and   (f) selecting, based on the fitness values for said n models, one of said n prediction rules as the basis for said final prediction rule.   
     
     
         16 . The computer readable medium of  claim 15 , further comprising computer executable instructions recorded thereon for determining a fitness value for said prediction rule by performing the method comprising summing a concordance index (CI) of said prediction rule with a product of a sensitivity and a specificity of said prediction rule. 
     
     
         17 . The computer readable medium of  claim 15 , further comprising computer executable instructions recorded thereon for determining a fitness value for said prediction rule by performing the method comprising measuring a separation between one or more outcome value(s) predicted by said prediction rule and actual outcome(s). 
     
     
         18 . The computer readable medium of  claim 15 , further comprising computer executable instructions recorded thereon for determining a fitness value for said prediction rule by performing the method comprising calculating the following risk functional:
     R ( k,l )=fitness/(1−sqrt(( k (log( l/k )+1)+log( l )/2)/ l ))   
       where l is a number of instances in said data for said set of n features, k is a VC-dimension of a set of functions from which said prediction rule is selected, and fitness is a function that evaluates quality of fit of said prediction rule. 
     
     
         19 . The computer readable medium of  claim 15 , further comprising computer executable instructions recorded thereon for determining a value of contribution by performing the method comprising determining a contribution value for each feature i by multiplying a weight of said feature i in said prediction rule with a measure of the discriminative ability of said feature i as observed in the data used to generate said prediction rule. 
     
     
         20 . The computer readable medium of  claim 15 , further comprising computer executable instructions recorded thereon for determining a value of contribution by performing the method comprising determining a contribution value for each feature i by multiplying a weight of said feature i in said prediction rule with a concordance index (CI) for said feature i.