Time-uniform conformal and PAC prediction

TL;DR

This paper extends conformal and PAC prediction methods to sequential data settings, providing anytime-valid uncertainty quantification with theoretical guarantees and practical demonstrations.

Contribution

It introduces a novel framework for sequential conformal and PAC prediction that maintains coverage guarantees without fixed sample sizes.

Findings

Methods are theoretically sound with proven coverage guarantees.

Validated on simulated datasets showing reliable uncertainty quantification.

Demonstrated utility on real-world streaming data.

Abstract

Given that machine learning algorithms are increasingly being deployed to aid in high stakes decision-making, uncertainty quantification methods that wrap around these black box models such as conformal prediction have received much attention in recent years. In sequential settings, where data are observed/generated in a streaming fashion, traditional conformal methods do not provide any guarantee without fixing the sample size. More importantly, traditional conformal methods cannot cope with sequentially updated predictions. As such, we develop an extension of the conformal prediction and related probably approximately correct (PAC) prediction frameworks to sequential settings where the number of data points is not fixed in advance. The resulting prediction sets are anytime-valid in that their expected coverage is at the required level at any time chosen by the analyst even if this choice depends on the data. We present theoretical guarantees for our proposed methods and demonstrate their validity and utility on simulated and real datasets.