Selective Classification Can Magnify Disparities Across Groups

TL;DR

Selective classification can improve accuracy but may also increase disparities between groups, especially with spurious correlations, and its effects depend on the margin distribution and full-coverage accuracy.

Contribution

This paper reveals how selective classification can magnify group disparities and introduces a theoretical framework based on margin distributions to understand this phenomenon.

Findings

Selective classification can worsen accuracy disparities across groups.

Increasing abstentions can decrease accuracy for some groups.

Training distributionally-robust models can mitigate disparity magnification.

Abstract

Selective classification, in which models can abstain on uncertain predictions, is a natural approach to improving accuracy in settings where errors are costly but abstentions are manageable. In this paper, we find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities between various groups within a population, especially in the presence of spurious correlations. We observe this behavior consistently across five vision and NLP datasets. Surprisingly, increasing abstentions can even decrease accuracies on some groups. To better understand this phenomenon, we study the margin distribution, which captures the model's confidences over all predictions. For symmetric margin distributions, we prove that whether selective classification monotonically improves or worsens accuracy is fully determined by the accuracy at full coverage (i.e., without any abstentions) and whether the distribution satisfies a property we call left-log-concavity. Our analysis also shows that selective classification tends to magnify full-coverage accuracy disparities. Motivated by our analysis, we train distributionally-robust models that achieve similar full-coverage accuracies across groups and show that selective classification uniformly improves each group on these models. Altogether, our results suggest that selective classification should be used with care and underscore the importance of training models to perform equally well across groups at full coverage.