Multi-model Ensemble Conformal Prediction in Dynamic Environments

TL;DR

This paper introduces a novel adaptive conformal prediction framework that dynamically selects from multiple models to produce more efficient prediction sets in changing environments while ensuring coverage guarantees.

Contribution

The paper presents a new adaptive conformal prediction method that dynamically chooses models to improve efficiency under distribution shifts, with proven regret bounds and empirical validation.

Findings

Outperforms existing methods in efficiency while maintaining coverage

Achieves strongly adaptive regret over all intervals

Works effectively on real and synthetic datasets

Abstract

Conformal prediction is an uncertainty quantification method that constructs a prediction set for a previously unseen datum, ensuring the true label is included with a predetermined coverage probability. Adaptive conformal prediction has been developed to address data distribution shifts in dynamic environments. However, the efficiency of prediction sets varies depending on the learning model used. Employing a single fixed model may not consistently offer the best performance in dynamic environments with unknown data distribution shifts. To address this issue, we introduce a novel adaptive conformal prediction framework, where the model used for creating prediction sets is selected on the fly from multiple candidate models. The proposed algorithm is proven to achieve strongly adaptive regret over all intervals while maintaining valid coverage. Experiments on real and synthetic datasets corroborate that the proposed approach consistently yields more efficient prediction sets while maintaining valid coverage, outperforming alternative methods.