Volume Optimality in Conformal Prediction with Structured Prediction Sets

TL;DR

This paper introduces a novel dynamic programming algorithm for conformal prediction sets, achieving near-optimal volume within a restricted family, and demonstrates its effectiveness through theoretical guarantees and experiments.

Contribution

The paper proves an impossibility result for volume optimality and proposes a DP-based method for near-optimal volume prediction sets within a finite VC-dimension family.

Findings

The method guarantees near-optimal volume for any distribution.

It outperforms existing methods in various experimental settings.

The approach extends to approximate conditional coverage.

Abstract

Conformal Prediction is a widely studied technique to construct prediction sets of future observations. Most conformal prediction methods focus on achieving the necessary coverage guarantees, but do not provide formal guarantees on the size (volume) of the prediction sets. We first prove an impossibility of volume optimality where any distribution-free method can only find a trivial solution. We then introduce a new notion of volume optimality by restricting the prediction sets to belong to a set family (of finite VC-dimension), specifically a union of $k$-intervals. Our main contribution is an efficient distribution-free algorithm based on dynamic programming (DP) to find a union of $k$-intervals that is guaranteed for any distribution to have near-optimal volume among all unions of $k$-intervals satisfying the desired coverage property. By adopting the framework of distributional conformal prediction (Chernozhukov et al., 2021), the new DP based conformity score can also be applied to achieve approximate conditional coverage and conditional restricted volume optimality, as long as a reasonable estimator of the conditional CDF is available. While the theoretical results already establish volume-optimality guarantees, they are complemented by experiments that demonstrate that our method can significantly outperform existing methods in many settings.