Exploring the Noise Robustness of Online Conformal Prediction

TL;DR

This paper investigates the robustness of online conformal prediction under label noise and proposes NR-OCP, a method that maintains coverage guarantees despite noisy labels, with theoretical and empirical validation.

Contribution

The paper introduces NR-OCP, a novel noise-robust online conformal prediction method that corrects coverage gaps caused by label noise using a robust pinball loss.

Findings

NR-OCP eliminates coverage gaps under label noise.

Theoretical convergence rate of 0T^{-1/2} for coverage errors.

Empirical results show improved coverage and efficiency.

Abstract

Conformal prediction is an emerging technique for uncertainty quantification that constructs prediction sets guaranteed to contain the true label with a predefined probability. Recent work develops online conformal prediction methods that adaptively construct prediction sets to accommodate distribution shifts. However, existing algorithms typically assume perfect label accuracy which rarely holds in practice. In this work, we investigate the robustness of online conformal prediction under uniform label noise with a known noise rate, in both constant and dynamic learning rate schedules. We show that label noise causes a persistent gap between the actual mis-coverage rate and the desired rate $\alpha$, leading to either overestimated or underestimated coverage guarantees. To address this issue, we propose Noise Robust Online Conformal Prediction (dubbed NR-OCP) by updating the threshold with a novel robust pinball loss, which provides an unbiased estimate of clean pinball loss without requiring ground-truth labels. Our theoretical analysis shows that NR-OCP eliminates the coverage gap in both constant and dynamic learning rate schedules, achieving a convergence rate of $\mathcal{O}(T^{-1/2})$ for both empirical and expected coverage errors under uniform label noise. Extensive experiments demonstrate the effectiveness of our method by achieving both precise coverage and improved efficiency.