Valid Selection among Conformal Sets

TL;DR

This paper introduces a stability-based method for selecting the best conformal prediction set, ensuring coverage guarantees are maintained even when choosing among multiple valid sets, with extensions to online settings and experimental validation.

Contribution

It proposes a novel stability-based approach for valid selection among conformal sets, extending to online scenarios and incorporating refinements with experimental demonstrations.

Findings

The method guarantees coverage after selection.

Effective in online conformal prediction settings.

Validated through comprehensive experiments.

Abstract

Conformal prediction offers a distribution-free framework for constructing prediction sets with coverage guarantees. In practice, multiple valid conformal prediction sets may be available, arising from different models or methodologies. However, selecting the most desirable set, such as the smallest, can invalidate the coverage guarantees. To address this challenge, we propose a stability-based approach that ensures coverage for the selected prediction set. We extend our results to the online conformal setting, propose several refinements in settings where additional structure is available, and demonstrate its effectiveness through experiments.