With Malice Towards None: Assessing Uncertainty via Equalized Coverage

TL;DR

This paper introduces a method called equalized coverage that constructs unbiased prediction intervals with equal coverage across protected groups, ensuring fairer decision-making in data-driven systems.

Contribution

The paper proposes a distribution-free, finite-sample guarantee methodology for unbiased prediction intervals that can be applied as a wrapper around any predictive algorithm.

Findings

Equalized coverage achieves unbiased prediction intervals across groups.

The method outperforms competitive approaches in real data tests.

It provides rigorous fairness guarantees in predictive uncertainty.

Abstract

An important factor to guarantee a fair use of data-driven recommendation systems is that we should be able to communicate their uncertainty to decision makers. This can be accomplished by constructing prediction intervals, which provide an intuitive measure of the limits of predictive performance. To support equitable treatment, we force the construction of such intervals to be unbiased in the sense that their coverage must be equal across all protected groups of interest. We present an operational methodology that achieves this goal by offering rigorous distribution-free coverage guarantees holding in finite samples. Our methodology, equalized coverage, is flexible as it can be viewed as a wrapper around any predictive algorithm. We test the applicability of the proposed framework on real data, demonstrating that equalized coverage constructs unbiased prediction intervals, unlike competitive methods.