Fair Decisions from Calibrated Scores: Achieving Optimal Classification While Satisfying Sufficiency

TL;DR

This paper develops an exact method for optimal binary classification under the fairness criterion of sufficiency, using group-calibrated scores, and provides a geometric framework for understanding achievable fairness-performance trade-offs.

Contribution

It introduces a geometric characterization and a simple post-processing algorithm for optimal classification under sufficiency with calibrated scores, addressing a gap in fairness-aware classification.

Findings

The algorithm achieves optimal classifiers using only group-calibrated scores and group membership.

It characterizes feasible PPV and FOR pairs for classifiers under sufficiency.

The method often matches the performance of the best classifiers that approximate separation.

Abstract

Binary classification based on predicted probabilities (scores) is a fundamental task in supervised machine learning. While thresholding scores is Bayes-optimal in the unconstrained setting, using a single threshold generally violates statistical group fairness constraints. Under independence (statistical parity) and separation (equalized odds), such thresholding suffices when the scores already satisfy the corresponding criterion. However, this does not extend to sufficiency: even perfectly group-calibrated scores -- including true class probabilities -- violate predictive parity after thresholding. In this work, we present an exact solution for optimal binary (randomized) classification under sufficiency, assuming finite sets of group-calibrated scores. We provide a geometric characterization of the feasible pairs of positive predictive value (PPV) and false omission rate (FOR) achievable by such classifiers, and use it to derive a simple post-processing algorithm that attains the optimal classifier using only group-calibrated scores and group membership. Finally, since sufficiency and separation are generally incompatible, we identify the classifier that minimizes deviation from separation subject to sufficiency, and show that it can also be obtained by our algorithm, often achieving performance comparable to the optimum.