Length Optimization in Conformal Prediction

TL;DR

This paper introduces CPL, a new conformal prediction framework that balances conditional validity with near-optimal set length, improving prediction efficiency across various data shifts and tasks.

Contribution

CPL is a novel framework that achieves length optimization while maintaining conditional validity under covariate shifts, with theoretical guarantees and empirical superiority.

Findings

CPL attains near-optimal prediction set length.

CPL ensures conditional validity under covariate shifts.

Empirical results show CPL outperforms existing methods.

Abstract

Conditional validity and length efficiency are two crucial aspects of conformal prediction (CP). Conditional validity ensures accurate uncertainty quantification for data subpopulations, while proper length efficiency ensures that the prediction sets remain informative. Despite significant efforts to address each of these issues individually, a principled framework that reconciles these two objectives has been missing in the CP literature. In this paper, we develop Conformal Prediction with Length-Optimization (CPL) - a novel and practical framework that constructs prediction sets with (near-) optimal length while ensuring conditional validity under various classes of covariate shifts, including the key cases of marginal and group-conditional coverage. In the infinite sample regime, we provide strong duality results which indicate that CPL achieves conditional validity and length optimality. In the finite sample regime, we show that CPL constructs conditionally valid prediction sets. Our extensive empirical evaluations demonstrate the superior prediction set size performance of CPL compared to state-of-the-art methods across diverse real-world and synthetic datasets in classification, regression, and large language model-based multiple choice question answering. An Implementation of our algorithm can be accessed at the following link: https://github.com/shayankiyani98/CP.