Fairness in machine learning: against false positive rate equality as a measure of fairness

TL;DR

This paper critically examines common fairness measures in machine learning, arguing that false positive rate equality does not meaningfully represent fairness and should not be used as a standard for evaluating algorithmic fairness.

Contribution

The paper provides an ethical analysis showing that false positive rate equality is an incoherent fairness standard, challenging its widespread use in machine learning.

Findings

False positive rate equality does not track a meaningful fairness property.

Calibration and false positive rate equality cannot both be satisfied simultaneously.

False positive rate equality sets an incoherent standard for fairness evaluation.

Abstract

As machine learning informs increasingly consequential decisions, different metrics have been proposed for measuring algorithmic bias or unfairness. Two popular fairness measures are calibration and equality of false positive rate. Each measure seems intuitively important, but notably, it is usually impossible to satisfy both measures. For this reason, a large literature in machine learning speaks of a fairness tradeoff between these two measures. This framing assumes that both measures are, in fact, capturing something important. To date, philosophers have not examined this crucial assumption, and examined to what extent each measure actually tracks a normatively important property. This makes this inevitable statistical conflict, between calibration and false positive rate equality, an important topic for ethics. In this paper, I give an ethical framework for thinking about these measures and argue that, contrary to initial appearances, false positive rate equality does not track anything about fairness, and thus sets an incoherent standard for evaluating the fairness of algorithms.