Controlling FSR in Selective Classification

TL;DR

This paper introduces an optimality function for selective classification that effectively controls false selection error rates in binary and multi-class scenarios, with proven theoretical guarantees and data-driven methods.

Contribution

It proposes a new optimality function for FSR control, extending its applicability from binary to multi-class classification and removing exchangeability assumptions.

Findings

The optimality function controls global FSR with finite samples.

The method extends from binary to multi-class classification.

FSR control is achievable without exchangeability assumptions.

Abstract

Uncertainty quantification and false selection error rate (FSR) control are crucial in many high-consequence scenarios, so we need models with good interpretability. This article introduces the optimality function for the binary classification problem in selective classification. We prove the optimality of this function in oracle situations and provide a data-driven method under the condition of exchangeability. We demonstrate it can control global FSR with the finite sample assumption and successfully extend the above situation from binary to multi-class classification. Furthermore, we demonstrate that FSR can still be controlled without exchangeability, ultimately completing the proof using the martingale method.