Selecting Informative Conformal Prediction Sets with an Optimized FCR-Controlled Approach

TL;DR

This paper develops an optimized approach for selecting informative conformal prediction sets that maintains false coverage rate control, improving power in classification tasks through oracle-guided policies and calibration.

Contribution

It introduces a new method for constructing conformal prediction sets with FCR control, leveraging oracle-guided policies and calibration to enhance power in practical settings.

Findings

Achieves higher power than existing methods.

Maintains finite sample FCR control through calibration.

Effective on both real and simulated data.

Abstract

Conformal methods provide prediction sets for outcomes with confidence guarantees. We study their use in a selective inference setting, where inference is performed only when the prediction set is informative. The analyst may consider as informative, for example, cases with prediction sets that are sufficiently small, exclude null values, or satisfy other appropriate monotone constraints. Because inference is typically restricted to informative cases in practical applications, accounting for the resulting selection bias is crucial to maintaining false coverage rate (FCR) control. A general framework for constructing such informative conformal prediction sets while controlling the FCR on the selected sample was suggested in Gazin et al. (2025). In this work we focus on oracle-guided procedures. We derive the optimal decision policy under a suitable power objective in the oracle setting where the probability of belonging to each prediction set can be computed. In practice, of course, only estimated probabilities are available. We therefore introduce a calibration procedure that adjusts the oracle policy to maintain finite sample FCR control. We show that this approach can achieve substantially higher power than available alternatives. We demonstrate the effectiveness of our new methods for classification outcomes on both real and simulated data.