Stable Localized Conformal Prediction via Transduction

TL;DR

This paper introduces Stable Conformal Prediction (StCP), a transfer learning method that enhances the stability of prediction sets in conformal prediction, especially with limited calibration data.

Contribution

It formalizes set stability in conformal prediction and proposes StCP, which improves stability using source and target data without extra labels.

Findings

StCP achieves more stable prediction sets than standard methods.

Theoretical analysis confirms StCP's coverage and stability properties.

Empirical results show improved stability with limited calibration data.

Abstract

Existing evaluations of conformal prediction, such as prediction efficiency and test-conditional coverage, are defined in expectation over the calibration data. In practice, when only one calibration set of limited size is available, prediction sets often exhibit high variability in size, especially for methods with localization. We formalize this concern as set stability, defined as the variance of the conditional expectation of the set size given the calibration data. To improve stability without requiring additional target-task labels, we propose Stable Conformal Prediction (StCP), a transfer learning approach that utilizes labeled source-task data and unlabeled target data. Theoretically, we characterize the marginal coverage and stability of StCP; empirically, it delivers more stable prediction sets than standard conformal prediction methods, especially for those with localization, when calibration data are limited.