Ask for More Than Bayes Optimal: A Theory of Indecisions for Classification

TL;DR

This paper develops a theoretical framework for selective classification that minimizes indecision while controlling error rates, enabling near-optimal accuracy even in challenging scenarios, with practical calibration methods and phase transition insights.

Contribution

It provides a complete characterization of minimax risk in selective classification, introduces a finite-sample calibration method, and uncovers phase transitions in indecision for Gaussian mixtures.

Findings

Minimizing indecision can achieve near-optimal accuracy.

Calibration methods with non-asymptotic guarantees are effective.

Phase transition in indecision occurs in Gaussian mixture models.

Abstract

Selective classification is a powerful tool for automated decision-making in high-risk scenarios, allowing classifiers to act only when confident and abstain when uncertainty is high. Given a target accuracy, our goal is to minimize indecisions, observations we do not automate. For difficult problems, the target accuracy may be unattainable without abstention. By using indecisions, we can control the misclassification rate to any user-specified level, even below the Bayes optimal error rate, while minimizing overall indecision mass. We provide a complete characterization of the minimax risk in selective classification, establishing continuity and monotonicity properties that enable optimal indecision selection. We revisit selective inference via the Neyman-Pearson testing framework, where indecision enables control of type 2 error given fixed type 1 error probability. For both classification and testing, we propose a finite-sample calibration method with non-asymptotic guarantees, proving plug-in classifiers remain consistent and that accuracy-based calibration effectively controls indecision mass. In the binary Gaussian mixture model, we uncover the first sharp phase transition in selective inference, showing minimal indecision can yield near-optimal accuracy even under poor class separation. Experiments on Gaussian mixtures and real datasets confirm that small indecision proportions yield substantial accuracy gains, making indecision a principled tool for risk control.