Quantifying and Improving Adaptivity in Conformal Prediction through Input Transformations

TL;DR

This paper introduces a new method for evaluating and improving adaptiveness in conformal prediction by using input transformations for better difficulty grouping, leading to more accurate and effective prediction sets.

Contribution

It proposes a novel binning method based on input transformations, new metrics for adaptiveness evaluation, and a group-conditional conformal prediction algorithm that enhances adaptiveness in practice.

Findings

New metrics correlate more strongly with true adaptiveness.

The proposed method outperforms existing approaches on ImageNet and medical tasks.

Input transformation-based binning improves difficulty estimation.

Abstract

Conformal prediction constructs a set of labels instead of a single point prediction, while providing a probabilistic coverage guarantee. Beyond the coverage guarantee, adaptiveness to example difficulty is an important property. It means that the method should produce larger prediction sets for more difficult examples, and smaller ones for easier examples. Existing evaluation methods for adaptiveness typically analyze coverage rate violation or average set size across bins of examples grouped by difficulty. However, these approaches often suffer from imbalanced binning, which can lead to inaccurate estimates of coverage or set size. To address this issue, we propose a binning method that leverages input transformations to sort examples by difficulty, followed by uniform-mass binning. Building on this binning, we introduce two metrics to better evaluate adaptiveness. These metrics provide more reliable estimates of coverage rate violation and average set size due to balanced binning, leading to more accurate adaptivity assessment. Through experiments, we demonstrate that our proposed metric correlates more strongly with the desired adaptiveness property compared to existing ones. Furthermore, motivated by our findings, we propose a new adaptive prediction set algorithm that groups examples by estimated difficulty and applies group-conditional conformal prediction. This allows us to determine appropriate thresholds for each group. Experimental results on both (a) an Image Classification (ImageNet) (b) a medical task (visual acuity prediction) show that our method outperforms existing approaches according to the new metrics.