Powerful batch conformal prediction for classification

TL;DR

This paper introduces a powerful conformal prediction method for classification that constructs valid batch prediction sets with improved efficiency by leveraging conformal p-values and the Simes inequality, applicable even under label shift.

Contribution

It proposes a novel approach for batch conformal prediction using conformal p-value combinations, offering increased power over Bonferroni correction and providing theoretical guarantees under various data assumptions.

Findings

The proposed method yields narrower prediction sets compared to Bonferroni correction.

The approach remains valid under label distribution shift.

Empirical results demonstrate improved performance on synthetic and real data.

Abstract

In a split conformal framework with $K$ classes, a calibration sample of $n$ labeled examples is observed for inference on the label of a new unlabeled example. We explore the setting where a `batch' of $m$ independent such unlabeled examples is given, and the goal is to construct a batch prediction set with 1-$\alpha$ coverage. Unlike individual prediction sets, the batch prediction set is a collection of label vectors of size $m$, while the calibration sample consists of univariate labels. A natural approach is to apply the Bonferroni correction, which concatenates individual prediction sets at level $1-\alpha/m$. We propose a uniformly more powerful solution, based on specific combinations of conformal $p$-values that exploit the Simes inequality. We provide a general recipe for valid inference with any combinations of conformal $p$-values, and compare the performance of several useful choices. Intuitively, the pooled evidence of relatively `easy' examples within the batch can help provide narrower batch prediction sets. Additionally, we introduce a more computationally intensive method that aggregates batch scores and can be even more powerful. The theoretical guarantees are established when all examples are independent and identically distributed (iid), as well as more generally when iid is assumed only conditionally within each class. Notably, our results remain valid under label distribution shift, since the distribution of the labels need not be the same in the calibration sample and in the new batch. The effectiveness of the methods is highlighted through illustrative synthetic and real data examples.