Estimation of Fair Ranking Metrics with Incomplete Judgments

TL;DR

This paper introduces a sampling and estimation method for fair ranking metrics that works effectively with limited protected attribute labels, enabling fair ranking evaluation in large-scale systems.

Contribution

It proposes a novel unbiased estimator for four fair ranking metrics that functions with very few labeled items, addressing data scarcity in fairness evaluation.

Findings

Estimator accurately predicts fair ranking metrics with limited labels

Method outperforms random annotation in robustness and reliability

Validated on both simulated and real-world datasets

Abstract

There is increasing attention to evaluating the fairness of search system ranking decisions. These metrics often consider the membership of items to particular groups, often identified using protected attributes such as gender or ethnicity. To date, these metrics typically assume the availability and completeness of protected attribute labels of items. However, the protected attributes of individuals are rarely present, limiting the application of fair ranking metrics in large scale systems. In order to address this problem, we propose a sampling strategy and estimation technique for four fair ranking metrics. We formulate a robust and unbiased estimator which can operate even with very limited number of labeled items. We evaluate our approach using both simulated and real world data. Our experimental results demonstrate that our method can estimate this family of fair ranking metrics and provides a robust, reliable alternative to exhaustive or random data annotation.