Control System for Learning to Rank Fairness

Abstract

A Bayesian test of demographic parity for learning to rank may be applied to determine ranking modifications. A fairness control system receiving a ranking of items may apply Bayes factors to determine a likelihood of bias for the ranking. These Bayes factors may include a factor for determining bias in each item and a factor for determining bias in the ranking of the items. An indicator of bias may be generated using the applied Bayes factors and the fairness control system may modify the ranking if the determines likelihood of bias satisfies modification criteria for the ranking.

1 .- 20 . (canceled) 
     
     
         21 . A computer-implemented method, comprising:
 receiving, from an external ranking system, a ranking of a plurality of items, the ranking comprising an amount of bias;   iteratively modifying the ranking until a likelihood of bias for the ranking does not exceed a modification criterion, wherein an iteration of modifying the ranking comprises:
 identifying a value of a multi-valued protected feature of the plurality of items subject to bias in exceeding the modification criterion; and 
 elevating a rank of an item of the plurality of items having the value of the multi-valued protected feature; and 
   outputting the modified ranking of the plurality of items.   
     
     
         22 . The computer-implemented method of  claim 21 , wherein the likelihood of bias of the plurality of items for the ranking of the plurality of items is determined with respect to the multi-valued protected feature. 
     
     
         23 . The computer-implemented method of  claim 21 , wherein the modification criterion comprises a threshold of demographic parity with respect to the multi-valued protected feature, and wherein demographic parity with respect to the multi-valued protected feature comprises ranking items of the plurality of items with a particular value of the feature proportional to a rate of occurrence of the particular value of the multi-valued protected feature relative to all values of the multi-valued protected feature. 
     
     
         24 . The computer-implemented method of  claim 21 , wherein the external ranking system is a ranking classifier trained using machine learning. 
     
     
         25 . The computer-implemented method of  claim 21 , wherein an iteration of modifying the ranking further comprises applying a Bayes factor to the ranking of the plurality of items to determine the likelihood of bias for the ranking of the items. 
     
     
         26 . The computer-implemented method of  claim 25 , wherein the Bayes factor comprises a ratio of a first metric to a second metric, wherein the first metric comprises a determination of a likelihood of bias of the plurality of items, and wherein the second metric comprises determination of a likelihood of absence of bias for the ranking. 
     
     
         27 . The computer-implemented method of  claim 21 , wherein an iteration of modifying the ranking further comprises applying respective Bayes factors to individual ones of the plurality of items to determine the likelihood of bias for the ranking of the items. 
     
     
         28 . One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors cause one or more computer systems to perform:
 receiving, from an external ranking system, a ranking of a plurality of items, the ranking comprising an amount of bias;   iteratively modifying the ranking until a likelihood of bias for the ranking does not exceed a modification criterion, wherein an iteration of modifying the ranking comprises:
 identifying a value of a multi-valued protected feature of the plurality of items subject to bias in exceeding the modification criterion; and 
 elevating a rank of an item of the plurality of items having the value of the multi-valued protected feature; and 
   outputting the modified ranking of the plurality of items.   
     
     
         29 . The one or more non-transitory computer-accessible storage media as recited in  claim 28 , wherein the likelihood of bias of the plurality of items for the ranking of the plurality of items is determined with respect to the multi-valued protected feature. 
     
     
         30 . The one or more non-transitory computer-accessible storage media as recited in  claim 28 , wherein the modification criterion comprises a threshold of demographic parity with respect to the multi-valued protected feature, and wherein demographic parity with respect to the multi-valued protected feature comprises ranking items of the plurality of items with a particular value of the feature proportional to a rate of occurrence of the particular value of the multi-valued protected feature relative to all values of the multi-valued protected feature. 
     
     
         31 . The one or more non-transitory computer-accessible storage media as recited in  claim 28 , wherein the external ranking system is a ranking classifier trained using machine learning. 
     
     
         32 . The one or more non-transitory computer-accessible storage media as recited in  claim 28 , wherein an iteration of modifying the ranking further comprises applying a Bayes factor to the ranking of the plurality of items to determine the likelihood of bias for the ranking of the items. 
     
     
         33 . The one or more non-transitory computer-accessible storage media as recited in  claim 32 , wherein the Bayes factor comprises a ratio of a first metric to a second metric, wherein the first metric comprises a determination of a likelihood of bias of the plurality of items, and wherein the second metric comprises determination of a likelihood of absence of bias for the ranking. 
     
     
         34 . The one or more non-transitory computer-accessible storage media as recited in  claim 28 , wherein an iteration of modifying the ranking further comprises applying respective Bayes factors to individual ones of the plurality of items to determine the likelihood of bias for the ranking of the items. 
     
     
         35 . A system, comprising:
 at least one processor;   a memory comprising program instructions that when executed by the at least one processor cause the at least one processor to implement a ranking modifier configured to:
 receive, from an external ranking system, a ranking of a plurality of items, the ranking comprising an amount of bias; 
 iteratively modify the ranking until a likelihood of bias for the ranking does not exceed a modification criterion, wherein to perform an iteration of modifying the ranking the ranking modifier is configured to:
 identify a value of a multi-valued protected feature of the plurality of items subject to bias in exceeding the modification criterion; and 
 elevate a rank of an item of the plurality of items having the value of the multi-valued protected feature; and 
 
 output the modified ranking of the plurality of items. 
   
     
     
         36 . The system of  claim 35 , wherein the likelihood of bias of the plurality of items for the ranking of the plurality of items is determined with respect to the multi-valued protected feature. 
     
     
         37 . The system of  claim 35 , wherein the modification criterion comprises a threshold of demographic parity with respect to the multi-valued protected feature, and wherein demographic parity with respect to the multi-valued protected feature comprises ranking items of the plurality of items with a particular value of the feature proportional to a rate of occurrence of the particular value of the multi-valued protected feature relative to all values of the multi-valued protected feature. 
     
     
         38 . The system of  claim 35 , wherein the external ranking system is a ranking classifier trained using machine learning. 
     
     
         39 . The system of  claim 35 , wherein an iteration of modifying the ranking further comprises applying a Bayes factor to the ranking of the plurality of items to determine the likelihood of bias for the ranking of the items. 
     
     
         40 . The system of  claim 39 , wherein the Bayes factor comprises a ratio of a first metric to a second metric, wherein the first metric comprises a determination of a likelihood of bias of the plurality of items, and wherein the second metric comprises determination of a likelihood of absence of bias for the ranking.