Training-Conditional Coverage Bounds under Covariate Shift

TL;DR

This paper develops bounds on the reliability of conformal prediction sets under covariate shift, providing PAC guarantees for training-conditional coverage that depend on distributional change severity and dataset size.

Contribution

It introduces the first bounds on training-conditional coverage for conformal methods under covariate shift, enhancing understanding of their reliability.

Findings

Provides upper bounds on the tail of training-conditional coverage distribution.

Characterizes how distributional change severity affects coverage reliability.

Offers PAC guarantees for conformal prediction under covariate shift.

Abstract

Conformal prediction methodology has recently been extended to the covariate shift setting, where the distribution of covariates differs between training and test data. While existing results ensure that the prediction sets from these methods achieve marginal coverage above a nominal level, their coverage rate conditional on the training dataset (referred to as training-conditional coverage) remains unexplored. In this paper, we address this gap by deriving upper bounds on the tail of the training-conditional coverage distribution, offering probably approximately correct (PAC) guarantees for these methods. Our results characterize the reliability of the prediction sets in terms of the severity of distributional changes and the size of the training dataset.