Anytime-Valid Conformal Risk Control
Bror Hultberg, Dave Zachariah, Ant\^onio H. Ribeiro

TL;DR
This paper introduces a method for conformal prediction that maintains valid error control with high probability over growing calibration datasets, even under distribution shifts, with theoretical guarantees and practical demonstrations.
Contribution
It extends conformal risk control to be valid at any time point with high probability over growing datasets, providing asymptotic tightness and applicability to distribution shift scenarios.
Findings
Guarantees high-probability error control over growing datasets
Establishes a matching lower bound for the guarantees
Demonstrates effectiveness through simulations and real data
Abstract
Prediction sets provide a means of quantifying the uncertainty in predictive tasks. Using held out calibration data, conformal prediction and risk control can produce prediction sets that exhibit statistically valid error control in a computationally efficient manner. However, in the standard formulations, the error is only controlled on average over many possible calibration datasets of fixed size. In this paper, we extend the control to remain valid with high probability over a cumulatively growing calibration dataset at any time point. We derive such guarantees using quantile-based arguments and illustrate the applicability of the proposed framework to settings involving distribution shift. We further establish a matching lower bound and show that our guarantees are asymptotically tight. Finally, we demonstrate the practical performance of our methods through both simulations and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Statistical Methods and Inference · Stochastic Gradient Optimization Techniques
