When Can We Reuse a Calibration Set for Multiple Conformal Predictions?

TL;DR

This paper proposes a method combining e-conformal prediction and Hoeffding's inequality to reuse a single calibration set multiple times in conformal prediction, maintaining coverage guarantees and improving practicality.

Contribution

It introduces a novel approach that allows multiple uses of one calibration set in conformal prediction by applying Hoeffding's inequality, reducing the need for repeated calibration.

Findings

Successful application on CIFAR-10 dataset

Maintains coverage guarantees with high probability

Reduces calibration data requirements

Abstract

Reliable uncertainty quantification is crucial for the trustworthiness of machine learning applications. Inductive Conformal Prediction (ICP) offers a distribution-free framework for generating prediction sets or intervals with user-specified confidence. However, standard ICP guarantees are marginal and typically require a fresh calibration set for each new prediction to maintain their validity. This paper addresses this practical limitation by demonstrating how e-conformal prediction, in conjunction with Hoeffding's inequality, can enable the repeated use of a single calibration set with a high probability of preserving the desired coverage. Through a case study on the CIFAR-10 dataset, we train a deep neural network and utilise a calibration set to estimate a Hoeffding correction. This correction allows us to apply a modified Markov's inequality, leading to the construction of prediction sets with quantifiable confidence. Our results illustrate the feasibility of maintaining provable performance in conformal prediction while enhancing its practicality by reducing the need for repeated calibration. The code for this work is publicly available.