WQLCP: Weighted Adaptive Conformal Prediction for Robust Uncertainty Quantification Under Distribution Shifts

TL;DR

This paper introduces WQLCP, a novel conformal prediction method that adaptively adjusts for distribution shifts, ensuring reliable uncertainty quantification with smaller prediction sets in real-world scenarios.

Contribution

The paper proposes WQLCP, an innovative weighted conformal prediction approach that improves coverage and reduces set size under distribution shifts, extending conformal prediction's applicability.

Findings

WQLCP maintains coverage under distribution shifts.

WQLCP produces smaller prediction sets compared to baselines.

Experiments on ImageNet variants validate robustness and efficiency.

Abstract

Conformal prediction (CP) provides a framework for constructing prediction sets with guaranteed coverage, assuming exchangeable data. However, real-world scenarios often involve distribution shifts that violate exchangeability, leading to unreliable coverage and inflated prediction sets. To address this challenge, we first introduce Reconstruction Loss-Scaled Conformal Prediction (RLSCP), which utilizes reconstruction losses derived from a Variational Autoencoder (VAE) as an uncertainty metric to scale score functions. While RLSCP demonstrates performance improvements, mainly resulting in better coverage, it quantifies quantiles based on a fixed calibration dataset without considering the discrepancies between test and train datasets in an unexchangeable setting. In the next step, we propose Weighted Quantile Loss-scaled Conformal Prediction (WQLCP), which refines RLSCP by incorporating a weighted notion of exchangeability, adjusting the calibration quantile threshold based on weights with respect to the ratio of calibration and test loss values. This approach improves the CP-generated prediction set outputs in the presence of distribution shifts. Experiments on large-scale datasets, including ImageNet variants, demonstrate that WQLCP outperforms existing baselines by consistently maintaining coverage while reducing prediction set sizes, providing a robust solution for CP under distribution shifts.