Conformal Prediction of Classifiers with Many Classes based on Noisy Labels

TL;DR

This paper introduces Noise-Aware Conformal Prediction (NACP), a method for calibrating conformal predictors with noisy labels, providing coverage guarantees even with many classes and noisy data.

Contribution

It proposes a novel approach to conformal prediction calibration using noisy labels, with theoretical guarantees and practical validation on image datasets.

Findings

Effective noise estimation for conformal thresholds

Finite sample coverage guarantees under uniform noise

Strong performance on large-class image datasets

Abstract

Conformal Prediction (CP) controls the prediction uncertainty of classification systems by producing a small prediction set, ensuring a predetermined probability that the true class lies within this set. This is commonly done by defining a score, based on the model predictions, and setting a threshold on this score using a validation set. In this study, we address the problem of CP calibration when we only have access to a calibration set with noisy labels. We show how we can estimate the noise-free conformal threshold based on the noisy labeled data. We derive a finite sample coverage guarantee for uniform noise that remains effective even in tasks with a large number of classes. We dub our approach Noise-Aware Conformal Prediction (NACP). We illustrate the performance of the proposed results on several standard image classification datasets with a large number of classes.