Generalization and Informativeness of Weighted Conformal Risk Control Under Covariate Shift

TL;DR

This paper investigates how weighted conformal risk control (W-CRC) maintains reliable prediction sets under covariate shift, providing theoretical bounds on its efficiency based on training data properties.

Contribution

It derives a bound on W-CRC's inefficiency under covariate shift, linking prediction set informativeness to training data and shift extent, with validation through experiments.

Findings

Bound on W-CRC inefficiency depends on hyperparameters and data quantities

Prediction set informativeness relates to covariate shift magnitude

Experimental validation confirms theoretical insights

Abstract

Predictive models are often required to produce reliable predictions under statistical conditions that are not matched to the training data. A common type of training-testing mismatch is covariate shift, where the conditional distribution of the target variable given the input features remains fixed, while the marginal distribution of the inputs changes. Weighted conformal risk control (W-CRC) uses data collected during the training phase to convert point predictions into prediction sets with valid risk guarantees at test time despite the presence of a covariate shift. However, while W-CRC provides statistical reliability, its efficiency -- measured by the size of the prediction sets -- can only be assessed at test time. In this work, we relate the generalization properties of the base predictor to the efficiency of W-CRC under covariate shifts. Specifically, we derive a bound on the inefficiency of the W-CRC predictor that depends on algorithmic hyperparameters and task-specific quantities available at training time. This bound offers insights on relationships between the informativeness of the prediction sets, the extent of the covariate shift, and the size of the calibration and training sets. Experiments on fingerprinting-based localization validate the theoretical results.