Conformal Predictive Systems Under Covariate Shift

TL;DR

This paper extends Conformal Predictive Systems to handle covariate shifts by introducing Weighted CPS, which uses likelihood ratios to maintain calibrated predictive distributions in non-IID scenarios, supported by theoretical and empirical evidence.

Contribution

It proposes Weighted CPS, a novel extension of CPS that accounts for covariate shifts using likelihood ratios, broadening the applicability of conformal prediction.

Findings

WCPS are probabilistically calibrated under covariate shift.

Theoretical foundations support WCPS validity and efficacy.

Empirical results on synthetic and real datasets demonstrate WCPS performance.

Abstract

Conformal Predictive Systems (CPS) offer a versatile framework for constructing predictive distributions, allowing for calibrated inference and informative decision-making. However, their applicability has been limited to scenarios adhering to the Independent and Identically Distributed (IID) model assumption. This paper extends CPS to accommodate scenarios characterized by covariate shifts. We therefore propose Weighted CPS (WCPS), akin to Weighted Conformal Prediction (WCP), leveraging likelihood ratios between training and testing covariate distributions. This extension enables the construction of nonparametric predictive distributions capable of handling covariate shifts. We present theoretical underpinnings and conjectures regarding the validity and efficacy of WCPS and demonstrate its utility through empirical evaluations on both synthetic and real-world datasets. Our simulation experiments indicate that WCPS are probabilistically calibrated under covariate shift.