Multi-model Online Conformal Prediction with Graph-Structured Feedback

TL;DR

This paper introduces a novel multi-model online conformal prediction algorithm that adaptively selects effective models using feedback, reducing computational complexity and prediction set size while maintaining coverage guarantees.

Contribution

The paper proposes a new algorithm that identifies effective models via feedback, improving efficiency and prediction set size in multi-model online conformal prediction.

Findings

Constructs smaller prediction sets compared to existing methods.

Ensures valid coverage with sublinear regret.

Outperforms existing approaches on real and synthetic datasets.

Abstract

Online conformal prediction has demonstrated its capability to construct a prediction set for each incoming data point that covers the true label with a predetermined probability. To cope with potential distribution shift, multi-model online conformal prediction has been introduced to select and leverage different models from a preselected candidate set. Along with the improved flexibility, the choice of the preselected set also brings challenges. A candidate set that includes a large number of models may increase the computational complexity. In addition, the inclusion of irrelevant models with poor performance may negatively impact the performance and lead to unnecessarily large prediction sets. To address these challenges, we propose a novel multi-model online conformal prediction algorithm that identifies a subset of effective models at each time step by collecting feedback from a bipartite graph, which is refined upon receiving new data. A model is then selected from this subset to construct the prediction set, resulting in reduced computational complexity and smaller prediction sets. Additionally, we demonstrate that using prediction set size as feedback, alongside model loss, can significantly improve efficiency by constructing smaller prediction sets while still satisfying the required coverage guarantee. The proposed algorithms are proven to ensure valid coverage and achieve sublinear regret. Experiments on real and synthetic datasets validate that the proposed methods construct smaller prediction sets and outperform existing multi-model online conformal prediction approaches.