A Simple, Statistically Robust Test of Discrimination

TL;DR

This paper introduces a hybrid statistical test combining benchmark and outcome tests to reliably detect discrimination in observational studies, proving that at least one test is correct under certain assumptions and demonstrating its effectiveness in real-world data.

Contribution

It presents a new hybrid testing method that guarantees correct conclusions when both component tests agree, improving robustness over traditional methods.

Findings

The hybrid test is robust to moderate assumption violations.

Application to California police stops reveals widespread racial discrimination.

Empirical validation shows the assumption holds in lending, education, and criminal justice.

Abstract

In observational studies of discrimination, the most common statistical approaches consider either the rate at which decisions are made (benchmark tests) or the success rate of those decisions (outcome tests). Both tests, however, have well-known statistical limitations, sometimes suggesting discrimination even when there is none. Despite the fallibility of the benchmark and outcome tests individually, here we prove a surprisingly strong statistical guarantee: under a common non-parametric assumption, at least one of the two tests must be correct; consequently, when both tests agree, they are guaranteed to yield correct conclusions. We present empirical evidence that the underlying assumption holds approximately in several important domains, including lending, education, and criminal justice -- and that our hybrid test is robust to the moderate violations of the assumption that we observe in practice. Applying this approach to 2.8 million police stops across California, we find evidence of widespread racial discrimination.