Testing Fairness with Utility Tradeoffs: A Wasserstein Projection Approach

TL;DR

This paper introduces a Wasserstein projection-based hypothesis testing framework to evaluate fairness and utility tradeoffs in data-driven decision-making, allowing for relaxed fairness constraints while maintaining utility thresholds.

Contribution

It presents a novel statistical testing method for fairness-utility tradeoffs using Wasserstein projections, applicable across various models and datasets.

Findings

The test effectively distinguishes intrinsic tradeoffs from data randomness.

Framework is computationally efficient and interpretable.

Applied to real datasets, revealing insights into fairness-utility balances.

Abstract

Ensuring fairness in data driven decision making has become a central concern across domains such as marketing, lending, and healthcare, but fairness constraints often come at the cost of utility. We propose a statistical hypothesis testing framework that jointly evaluates approximate fairness and utility, relaxing strict fairness requirements while ensuring that overall utility remains above a specified threshold. Our framework builds on the strong demographic parity (SDP) criterion and incorporates a utility measure motivated by the potential outcomes framework. The test statistic is constructed via Wasserstein projections, enabling auditors to assess whether observed fairness-utility tradeoffs are intrinsic to the algorithm or attributable to randomness in the data. We show that the test is computationally tractable, interpretable, broadly applicable across machine learning models, and extendable to more general settings. We apply our approach to multiple real-world datasets, offering new insights into the fairness-utility tradeoff through the perspective of statistical hypothesis testing.