Non-exchangeable Conformal Prediction with Optimal Transport: Tackling Distribution Shifts with Unlabeled Data

TL;DR

This paper introduces a novel approach using optimal transport to adapt conformal prediction methods for distribution shifts, providing a principled way to estimate and mitigate coverage loss without prior shift information.

Contribution

It proposes a new framework that leverages optimal transport to handle arbitrary distribution shifts in conformal prediction, extending its applicability beyond exchangeable data.

Findings

Effective estimation of coverage loss under distribution shifts

Mitigation of coverage loss without prior shift knowledge

Broad applicability to real-world non-exchangeable data

Abstract

Conformal prediction is a distribution-free uncertainty quantification method that has gained popularity in the machine learning community due to its finite-sample guarantees and ease of use. Its most common variant, dubbed split conformal prediction, is also computationally efficient as it boils down to collecting statistics of the model predictions on some calibration data not yet seen by the model. Nonetheless, these guarantees only hold if the calibration and test data are exchangeable, a condition that is difficult to verify and often violated in practice due to so-called distribution shifts. The literature is rife with methods to mitigate the loss in coverage in this non-exchangeable setting, but these methods require some prior information on the type of distribution shift to be expected at test time. In this work, we study this problem via a new perspective, through the lens of optimal transport, and show that it is possible to estimate the loss in coverage and mitigate arbitrary distribution shifts, offering a principled and broadly applicable solution.