Confidence Intervals for Testing Disparate Impact in Fair Learning

TL;DR

This paper derives the asymptotic distribution of key metrics for assessing disparate impact in machine learning, emphasizing the importance of confidence intervals over single-point estimates for fair testing.

Contribution

It introduces the asymptotic distribution of disparity metrics and advocates for confidence interval usage in testing group disparate impact.

Findings

Confidence intervals provide more reliable testing of disparate impact.

Asymptotic distribution formulas enable better statistical inference.

Illustrative examples demonstrate the importance of confidence intervals.

Abstract

We provide the asymptotic distribution of the major indexes used in the statistical literature to quantify disparate treatment in machine learning. We aim at promoting the use of confidence intervals when testing the so-called group disparate impact. We illustrate on some examples the importance of using confidence intervals and not a single value.