Quantifying Epistemic Predictive Uncertainty in Conformal Prediction

TL;DR

This paper introduces a method to quantify epistemic predictive uncertainty within conformal prediction by analyzing the credal set of predictive distributions, leading to more informative uncertainty assessments.

Contribution

It proves that conformal prediction regions correspond to the intersection of all distributions in the credal set and proposes a new uncertainty measure based on Maximum Mean Imprecision.

Findings

EPU quantification improves active learning and classification decisions.

The proposed measure captures conflicting information within the credal set.

Conformal prediction serves as a principled basis for epistemic uncertainty estimation.

Abstract

We study the problem of quantifying epistemic predictive uncertainty (EPU) -- that is, uncertainty faced at prediction time due to the existence of multiple plausible predictive models -- within the framework of conformal prediction (CP). To expose the implicit model multiplicity underlying CP, we build on recent results showing that, under a mild assumption, any full CP procedure induces a set of closed and convex predictive distributions, commonly referred to as a credal set. Importantly, the conformal prediction region (CPR) coincides exactly with the set of labels to which all distributions in the induced credal set assign probability at least $1-\alpha$. As our first contribution, we prove that this characterisation also holds in split CP. Building on this connection, we then propose a computationally efficient and analytically tractable uncertainty measure, based on \emph{Maximum Mean Imprecision}, to quantify the EPU by measuring the degree of conflicting information within the induced credal set. Experiments on active learning and selective classification demonstrate that the quantified EPU provides substantially more informative and fine-grained uncertainty assessments than reliance on CPR size alone. More broadly, this work highlights the potential of CP serving as a principled basis for decision-making under epistemic uncertainty.