Inference for an Algorithmic Fairness-Accuracy Frontier

TL;DR

This paper introduces a new statistical method to evaluate and compare algorithms based on their fairness and accuracy trade-offs, providing tools for hypothesis testing and identifying less discriminatory options.

Contribution

It develops a debiased machine learning estimator for the fairness-accuracy frontier, enabling rigorous inference and comparison of algorithms in terms of fairness and accuracy.

Findings

The proposed estimator performs well in finite samples.

Application to hospital algorithms shows potential for fairness improvements.

Framework identifies algorithms on the fairness-accuracy frontier.

Abstract

Algorithms are increasingly used to aid with high-stakes decision making. Yet, their predictive ability frequently exhibits systematic variation across population subgroups. To assess the trade-off between fairness and accuracy using finite data, we propose a debiased machine learning estimator for the fairness-accuracy frontier introduced by Liang, Lu, Mu, and Okumura (2024). We derive its asymptotic distribution and propose inference methods to test key hypotheses in the fairness literature, such as (i) whether excluding group identity from use in training the algorithm is optimal and (ii) whether there are less discriminatory alternatives to a given algorithm. In addition, we construct an estimator for the distance between a given algorithm and the fairest point on the frontier, and characterize its asymptotic distribution. Using Monte Carlo simulations, we evaluate the finite-sample performance of our inference methods. We apply our framework to re-evaluate algorithms used in hospital care management and show that our approach yields alternative algorithms that lie on the fairness-accuracy frontier, offering improvements along both dimensions.