Test-time Recalibration of Conformal Predictors Under Distribution Shift Based on Unlabeled Examples

TL;DR

This paper introduces a method for recalibrating conformal predictors using unlabeled data to maintain reliable uncertainty estimates under distribution shifts, addressing a key challenge in deploying classifiers in real-world scenarios.

Contribution

The authors propose a novel approach for cutoff threshold prediction using unlabeled examples, with theoretical guarantees under certain distribution shift models.

Findings

Method achieves accurate uncertainty estimates under natural distribution shifts.

Provably effective for specific models of distribution shift.

Addresses calibration challenges without requiring labeled data from new distributions.

Abstract

Modern image classifiers are very accurate, but the predictions come without uncertainty estimates. Conformal predictors provide uncertainty estimates by computing a set of classes containing the correct class with a user-specified probability based on the classifier's probability estimates. To provide such sets, conformal predictors often estimate a cutoff threshold for the probability estimates based on a calibration set. Conformal predictors guarantee reliability only when the calibration set is from the same distribution as the test set. Therefore, conformal predictors need to be recalibrated for new distributions. However, in practice, labeled data from new distributions is rarely available, making calibration infeasible. In this work, we consider the problem of predicting the cutoff threshold for a new distribution based on unlabeled examples. While it is impossible in general to guarantee reliability when calibrating based on unlabeled examples, we propose a method that provides excellent uncertainty estimates under natural distribution shifts, and provably works for a specific model of a distribution shift.