Online selective conformal inference: adaptive scores, convergence rate and optimality

TL;DR

This paper introduces OnlineSCI, an adaptive online conformal inference method that controls coverage and error rates, with proven convergence and optimality properties, applicable to various online selective tasks.

Contribution

The paper extends conformal inference to an adaptive online setting with selective inference capabilities, providing convergence rates and optimality guarantees.

Findings

OnlineSCI controls average missed coverage in adversarial settings.

It also controls instantaneous error rates at selected times.

Numerical experiments demonstrate practical effectiveness.

Abstract

In a supervised online setting, quantifying uncertainty has been proposed in the seminal work of \cite{gibbs2021adaptive}. For any given point-prediction algorithm, their method (ACI) produces a conformal prediction set with an average missed coverage getting close to a pre-specified level $\alpha$ for a long time horizon. We introduce an extended version of this algorithm, called OnlineSCI, allowing the user to additionally select times where such an inference should be made. OnlineSCI encompasses several prominent online selective tasks, such as building prediction intervals for extreme outcomes, classification with abstention, and online testing. While OnlineSCI controls the average missed coverage on the selected in an adversarial setting, our theoretical results also show that it controls the instantaneous error rate (IER) at the selected times, up to a non-asymptotical remainder term. Importantly, our theory covers the case where OnlineSCI updates the point-prediction algorithm at each time step, a property which we refer to as {\it adaptive} capability. We show that the adaptive versions of OnlineSCI can convergence to an optimal solution and provide an explicit convergence rate in each of the aforementioned application cases, under specific mild conditions. Finally, the favorable behavior of OnlineSCI in practice is illustrated by numerical experiments.