Conformal Prediction with Corrupted Labels: Uncertain Imputation and Robust Re-weighting

TL;DR

This paper develops robust conformal prediction methods for datasets with corrupted labels, introducing uncertain imputation and re-weighting techniques that maintain valid uncertainty quantification despite data corruptions.

Contribution

It proposes new conformal prediction methods that handle corrupted labels using uncertain imputation and robust re-weighting, with theoretical guarantees and empirical validation.

Findings

Robust conformal prediction remains valid with poorly estimated weights.

Uncertain imputation effectively preserves label uncertainty.

The triply robust framework guarantees valid predictions under multiple conditions.

Abstract

We introduce a framework for robust uncertainty quantification in situations where labeled training data are corrupted, through noisy or missing labels. We build on conformal prediction, a statistical tool for generating prediction sets that cover the test label with a pre-specified probability. The validity of conformal prediction, however, holds under the i.i.d assumption, which does not hold in our setting due to the corruptions in the data. To account for this distribution shift, the privileged conformal prediction (PCP) method proposed leveraging privileged information (PI) -- additional features available only during training -- to re-weight the data distribution, yielding valid prediction sets under the assumption that the weights are accurate. In this work, we analyze the robustness of PCP to inaccuracies in the weights. Our analysis indicates that PCP can still yield valid uncertainty estimates even when the weights are poorly estimated. Furthermore, we introduce uncertain imputation (UI), a new conformal method that does not rely on weight estimation. Instead, we impute corrupted labels in a way that preserves their uncertainty. Our approach is supported by theoretical guarantees and validated empirically on both synthetic and real benchmarks. Finally, we show that these techniques can be integrated into a triply robust framework, ensuring statistically valid predictions as long as at least one underlying method is valid.