What's in a Query: Polarity-Aware Distribution-Based Fair Ranking

TL;DR

This paper introduces a divergence-based framework for fair ranking that considers attention and relevance over time, ensuring more reliable fairness measures in search systems.

Contribution

It proposes new divergence-based fairness measures for amortized ranking, establishes bounds between individual and group fairness, and highlights the importance of query information in fair ranking.

Findings

New divergence-based fairness measures (DistFaiR) for ranking.

Group fairness is upper-bounded by individual fairness under these measures.

Ignoring query information can lead to fairwashing in practice.

Abstract

Machine learning-driven rankings, where individuals (or items) are ranked in response to a query, mediate search exposure or attention in a variety of safety-critical settings. Thus, it is important to ensure that such rankings are fair. Under the goal of equal opportunity, attention allocated to an individual on a ranking interface should be proportional to their relevance across search queries. In this work, we examine amortized fair ranking -- where relevance and attention are cumulated over a sequence of user queries to make fair ranking more feasible in practice. Unlike prior methods that operate on expected amortized attention for each individual, we define new divergence-based measures for attention distribution-based fairness in ranking (DistFaiR), characterizing unfairness as the divergence between the distribution of attention and relevance corresponding to an individual over time. This allows us to propose new definitions of unfairness, which are more reliable at test time. Second, we prove that group fairness is upper-bounded by individual fairness under this definition for a useful class of divergence measures, and experimentally show that maximizing individual fairness through an integer linear programming-based optimization is often beneficial to group fairness. Lastly, we find that prior research in amortized fair ranking ignores critical information about queries, potentially leading to a fairwashing risk in practice by making rankings appear more fair than they actually are.