Confidence on the Focal: Conformal Prediction with Selection-Conditional Coverage

TL;DR

This paper develops a framework for conformal prediction that guarantees valid coverage for selected units, addressing the bias introduced by data-driven selection in practical uncertainty quantification.

Contribution

It introduces a general, permutation-invariant method for constructing valid prediction sets conditional on selection, extending conformal prediction to multiple units and complex selection rules.

Findings

Framework achieves finite-sample exact coverage for selected units.

Method is computationally efficient for various realistic selection rules.

Applications demonstrate improved uncertainty quantification in drug discovery and health risk prediction.

Abstract

Conformal prediction builds marginally valid prediction intervals that cover the unknown outcome of a randomly drawn test point with a prescribed probability. However, in practice, data-driven methods are often used to identify specific test unit(s) of interest, requiring uncertainty quantification tailored to these focal units. In such cases, marginally valid conformal prediction intervals may fail to provide valid coverage for the focal unit(s) due to selection bias. This paper presents a general framework for constructing a prediction set with finite-sample exact coverage, conditional on the unit being selected by a given procedure. The general form of our method accommodates arbitrary selection rules that are invariant to the permutation of the calibration units, and generalizes Mondrian Conformal Prediction to multiple test units and non-equivariant classifiers. We also work out computationally efficient implementation of our framework for a number of realistic selection rules, including top-K selection, optimization-based selection, selection based on conformal p-values, and selection based on properties of preliminary conformal prediction sets. The performance of our methods is demonstrated via applications in drug discovery and health risk prediction.