Does calibration mean what they say it means; or, the reference class problem rises again

TL;DR

The paper argues that calibration does not guarantee consistent interpretation of risk scores across groups and highlights the reference class problem as a fundamental issue in fairness criteria.

Contribution

It critically examines the assumption that calibration ensures fairness and reveals the pervasive influence of the reference class problem in fairness metrics.

Findings

Calibration cannot ensure consistent score interpretation across groups

The reference class fallacy undermines fairness claims based on calibration

The reference class problem affects multiple group fairness criteria

Abstract

Discussions of statistical criteria for fairness commonly convey the normative significance of calibration within groups by invoking what risk scores "mean." On the Same Meaning picture, group-calibrated scores "mean the same thing" (on average) across individuals from different groups and accordingly, guard against disparate treatment of individuals based on group membership. My contention is that calibration guarantees no such thing. Since concrete actual people belong to many groups, calibration cannot ensure the kind of consistent score interpretation that the Same Meaning picture implies matters for fairness, unless calibration is met within every group to which an individual belongs. Alas only perfect predictors may meet this bar. The Same Meaning picture thus commits a reference class fallacy by inferring from calibration within some group to the "meaning" or evidential value of an individual's score, because they are a member of that group. The reference class answer it presumes does not only lack justification; it is very likely wrong. I then show that the reference class problem besets not just calibration but other group statistical criteria that claim a close connection to fairness. Reflecting on the origins of this oversight opens a wider lens onto the predominant methodology in algorithmic fairness based on stylized cases.