Selective conformal inference with false coverage-statement rate control

TL;DR

This paper introduces SCOP, a novel conformal inference method that controls the false coverage-statement rate in post-selection scenarios, providing valid and narrower prediction intervals compared to existing approaches.

Contribution

The paper develops SCOP, a new selective conformal inference framework that achieves exact FCR control and offers non-asymptotic bounds, improving over previous methods.

Findings

SCOP effectively controls FCR in various settings.

Numerical results show narrower prediction intervals with SCOP.

SCOP outperforms existing methods in robustness and efficiency.

Abstract

Conformal inference is a popular tool for constructing prediction intervals (PI). We consider here the scenario of post-selection/selective conformal inference, that is PIs are reported only for individuals selected from an unlabeled test data. To account for multiplicity, we develop a general split conformal framework to construct selective PIs with the false coverage-statement rate (FCR) control. We first investigate the Benjamini and Yekutieli (2005)'s FCR-adjusted method in the present setting, and show that it is able to achieve FCR control but yields uniformly inflated PIs. We then propose a novel solution to the problem, named as Selective COnditional conformal Predictions (SCOP), which entails performing selection procedures on both calibration set and test set and construct marginal conformal PIs on the selected sets by the aid of conditional empirical distribution obtained by the calibration set. Under a unified framework and exchangeable assumptions, we show that the SCOP can exactly control the FCR. More importantly, we provide non-asymptotic miscoverage bounds for a general class of selection procedures beyond exchangeablity and discuss the conditions under which the SCOP is able to control the FCR. As special cases, the SCOP with quantile-based selection or conformal p-values-based multiple testing procedures enjoys valid coverage guarantee under mild conditions. Numerical results confirm the effectiveness and robustness of SCOP in FCR control and show that it achieves more narrowed PIs over existing methods in many settings.