Selective Conformal Risk Control

TL;DR

This paper introduces Selective Conformal Risk Control (SCRC), a framework combining conformal prediction with selective classification to produce smaller, reliable prediction sets with finite-sample guarantees.

Contribution

The paper proposes a unified framework and two algorithms for selective conformal risk control, improving the practicality and efficiency of uncertainty quantification in high-stakes machine learning.

Findings

Both algorithms achieve target coverage and risk levels.

SCRC-I offers better computational efficiency with slightly more conservative risk.

Experiments demonstrate effective and reliable uncertainty quantification.

Abstract

Reliable uncertainty quantification is essential for deploying machine learning systems in high-stakes domains. Conformal prediction provides distribution-free coverage guarantees but often produces overly large prediction sets, limiting its practical utility. To address this issue, we propose \textit{Selective Conformal Risk Control} (SCRC), a unified framework that integrates conformal prediction with selective classification. The framework formulates uncertainty control as a two-stage problem: the first stage selects confident samples for prediction, and the second stage applies conformal risk control on the selected subset to construct calibrated prediction sets. We develop two algorithms under this framework. The first, SCRC-T, preserves exchangeability by computing thresholds jointly over calibration and test samples, offering exact finite-sample guarantees. The second, SCRC-I, is a calibration-only variant that provides PAC-style probabilistic guarantees while being more computational efficient. Experiments on two public datasets show that both methods achieve the target coverage and risk levels, with nearly identical performance, while SCRC-I exhibits slightly more conservative risk control but superior computational practicality. Our results demonstrate that selective conformal risk control offers an effective and efficient path toward compact, reliable uncertainty quantification.