Revisiting the Berkeley Admissions data: Statistical Tests for Causal Hypotheses

TL;DR

This paper reexamines the Berkeley admissions data through a causal framework, introducing statistical tests for causal fairness hypotheses and comparing different causal notions to better understand fairness in admissions.

Contribution

It introduces a causal hypothesis testing approach using Pearl's instrumental-variable inequalities and analyzes various causal fairness notions with observational data.

Findings

Statistical tests based on causal models can distinguish fairness hypotheses.

Different causal fairness notions may not be equivalent but can yield similar statistical tests.

A causal perspective clarifies the pitfalls of correlation-based fairness assessments.

Abstract

Reasoning about fairness through correlation-based notions is rife with pitfalls. The 1973 University of California, Berkeley graduate school admissions case from Bickel et. al. (1975) is a classic example of one such pitfall, namely Simpson's paradox. The discrepancy in admission rates among males and female applicants, in the aggregate data over all departments, vanishes when admission rates per department are examined. We reason about the Berkeley graduate school admissions case through a causal lens. In the process, we introduce a statistical test for causal hypothesis testing based on Pearl's instrumental-variable inequalities (Pearl 1995). We compare different causal notions of fairness that are based on graphical, counterfactual and interventional queries on the causal model, and develop statistical tests for these notions that use only observational data. We study the logical relations between notions, and show that while notions may not be equivalent, their corresponding statistical tests coincide for the case at hand. We believe that a thorough case-based causal analysis helps develop a more principled understanding of both causal hypothesis testing and fairness.