Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift

TL;DR

This paper investigates how pseudo-calibration can help maintain coverage guarantees in conformal prediction under distribution shift, providing theoretical bounds and a practical algorithm to improve robustness.

Contribution

It introduces a domain adaptation-based analysis of pseudo-calibration, deriving bounds and proposing a source-tuned algorithm to mitigate coverage loss under distribution shift.

Findings

Bounds qualitatively track pseudo-calibration behavior

Source-tuned scheme reduces coverage degradation

Maintains prediction set sizes under shift

Abstract

Conformal prediction (CP) offers distribution-free marginal coverage guarantees under an exchangeability assumption, but these guarantees can fail if the data distribution shifts. We analyze the use of pseudo-calibration as a tool to counter this performance loss under a bounded label-conditional covariate shift model. Using tools from domain adaptation, we derive a lower bound on target coverage in terms of the source-domain loss of the classifier and a Wasserstein measure of the shift. Using this result, we provide a method to design pseudo-calibrated sets that inflate the conformal threshold by a slack parameter to keep target coverage above a prescribed level. Finally, we propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and randomized labels as a function of classifier uncertainty. Numerical experiments show that our bounds qualitatively track pseudo-calibration behavior and that the source-tuned scheme mitigates coverage degradation under distribution shift while maintaining nontrivial prediction set sizes.