Stretching the Effectiveness of MLE from Accuracy to Bias for Pairwise Comparisons

TL;DR

This paper improves the fairness of estimators in pairwise comparison models by modifying the MLE to reduce bias while preserving its optimal accuracy in mean squared error.

Contribution

It introduces a simple modification to the MLE that reduces bias, enhancing fairness without sacrificing accuracy in pairwise comparison estimation.

Findings

Modified estimator reduces bias compared to MLE

Maintains minimax-optimality in mean squared error

Enhances fairness in pairwise comparison evaluations

Abstract

A number of applications (e.g., AI bot tournaments, sports, peer grading, crowdsourcing) use pairwise comparison data and the Bradley-Terry-Luce (BTL) model to evaluate a given collection of items (e.g., bots, teams, students, search results). Past work has shown that under the BTL model, the widely-used maximum-likelihood estimator (MLE) is minimax-optimal in estimating the item parameters, in terms of the mean squared error. However, another important desideratum for designing estimators is fairness. In this work, we consider fairness modeled by the notion of bias in statistics. We show that the MLE incurs a suboptimal rate in terms of bias. We then propose a simple modification to the MLE, which "stretches" the bounding box of the maximum-likelihood optimizer by a small constant factor from the underlying ground truth domain. We show that this simple modification leads to an improved rate in bias, while maintaining minimax-optimality in the mean squared error. In this manner, our proposed class of estimators provably improves fairness represented by bias without loss in accuracy.