Selective Classification Under Distribution Shifts

TL;DR

This paper introduces a new framework for selective classification that accounts for distribution shifts, including out-of-distribution and covariate shifts, enhancing reliability of deep learning classifiers in real-world scenarios.

Contribution

It proposes the first generalized selective classification framework addressing distribution shifts and introduces two novel margin-based confidence scores for deep learning classifiers.

Findings

Proposed score functions outperform existing methods in experiments.

Framework effectively handles label-shifted and covariate-shifted data.

Enhanced reliability of classifiers under distribution shifts.

Abstract

In selective classification (SC), a classifier abstains from making predictions that are likely to be wrong to avoid excessive errors. To deploy imperfect classifiers -- either due to intrinsic statistical noise of data or for robustness issue of the classifier or beyond -- in high-stakes scenarios, SC appears to be an attractive and necessary path to follow. Despite decades of research in SC, most previous SC methods still focus on the ideal statistical setting only, i.e., the data distribution at deployment is the same as that of training, although practical data can come from the wild. To bridge this gap, in this paper, we propose an SC framework that takes into account distribution shifts, termed generalized selective classification, that covers label-shifted (or out-of-distribution) and covariate-shifted samples, in addition to typical in-distribution samples, the first of its kind in the SC literature. We focus on non-training-based confidence-score functions for generalized SC on deep learning (DL) classifiers, and propose two novel margin-based score functions. Through extensive analysis and experiments, we show that our proposed score functions are more effective and reliable than the existing ones for generalized SC on a variety of classification tasks and DL classifiers. Code is available at https://github.com/sun-umn/sc_with_distshift.