Optimal training-conditional regret for online conformal prediction

TL;DR

This paper develops and analyzes algorithms for online conformal prediction in non-stationary data streams, providing minimax-optimal regret guarantees under distribution drift, with practical algorithms for both pretrained and online-trained non-conformity scores.

Contribution

It introduces adaptive conformal prediction algorithms that handle distribution drift using drift detection, achieving minimax-optimal regret guarantees in non-stationary environments.

Findings

Algorithms achieve minimax-optimal regret bounds.

Numerical experiments confirm theoretical guarantees.

Handles both abrupt change points and smooth drift.

Abstract

We study online conformal prediction for non-stationary data streams subject to unknown distribution drift. While most prior work studied this problem under adversarial settings and/or assessed performance in terms of gaps of time-averaged marginal coverage, we instead evaluate performance through training-conditional cumulative regret. We specifically focus on independently generated data with two types of distribution shift: abrupt change points and smooth drift. When non-conformity score functions are pretrained on an independent dataset, we propose a split-conformal style algorithm that leverages drift detection to adaptively update calibration sets, which provably achieves minimax-optimal regret. When non-conformity scores are instead trained online, we develop a full-conformal style algorithm that again incorporates drift detection to handle non-stationarity; this approach relies on stability - rather than permutation symmetry - of the model-fitting algorithm, which is often better suited to online learning under evolving environments. We establish non-asymptotic regret guarantees for our online full conformal algorithm, which match the minimax lower bound under appropriate restrictions on the prediction sets. Numerical experiments corroborate our theoretical findings.