Test-time augmentation improves efficiency in conformal prediction

TL;DR

This paper demonstrates that test-time augmentation significantly reduces the size of prediction sets in conformal classifiers, improving efficiency without retraining across various datasets and models.

Contribution

It introduces a flexible, efficient TTA method that reduces conformal prediction set sizes by 10-14% without retraining, applicable to any conformal score.

Findings

Test-time augmentation reduces conformal set sizes by 10-14%.

The approach is effective across multiple datasets, models, and conformal scoring methods.

TTA improves efficiency under different distribution shifts.

Abstract

A conformal classifier produces a set of predicted classes and provides a probabilistic guarantee that the set includes the true class. Unfortunately, it is often the case that conformal classifiers produce uninformatively large sets. In this work, we show that test-time augmentation (TTA)--a technique that introduces inductive biases during inference--reduces the size of the sets produced by conformal classifiers. Our approach is flexible, computationally efficient, and effective. It can be combined with any conformal score, requires no model retraining, and reduces prediction set sizes by 10%-14% on average. We conduct an evaluation of the approach spanning three datasets, three models, two established conformal scoring methods, different guarantee strengths, and several distribution shifts to show when and why test-time augmentation is a useful addition to the conformal pipeline.