The Relationship between No-Regret Learning and Online Conformal Prediction

TL;DR

This paper explores the connection between no-regret learning and online conformal prediction, revealing how certain algorithms can guarantee group-conditional coverage in adversarial environments, extending prior marginal coverage results.

Contribution

It establishes a tight link between swap-regret and group-conditional coverage, and shows how follow-the-perturbed-leader algorithms can ensure coverage guarantees in adversarial settings.

Findings

Threshold calibrated coverage relates to swap-regret in adversarial settings.

Follow-the-perturbed-leader algorithms provide group-conditional coverage guarantees.

Experimental results demonstrate the effectiveness of the proposed methods.

Abstract

Existing algorithms for online conformal prediction -- guaranteeing marginal coverage in adversarial settings -- are variants of online gradient descent (OGD), but their analyses of worst-case coverage do not follow from the regret guarantee of OGD. What is the relationship between no-regret learning and online conformal prediction? We observe that although standard regret guarantees imply marginal coverage in i.i.d. settings, this connection fails as soon as we either move to adversarial environments or ask for group conditional coverage. On the other hand, we show a tight connection between threshold calibrated coverage and swap-regret in adversarial settings, which extends to group-conditional (multi-valid) coverage. We also show that algorithms in the follow the perturbed leader family of no regret learning algorithms (which includes online gradient descent) can be used to give group-conditional coverage guarantees in adversarial settings for arbitrary grouping functions. Via this connection we analyze and conduct experiments using a multi-group generalization of the ACI algorithm of Gibbs & Candes [2021] (arXiv:2106.00170).