Integral Imprecise Probability Metrics

TL;DR

This paper introduces the integral imprecise probability metric framework, a novel approach based on Choquet integrals for comparing and quantifying epistemic uncertainty in imprecise probability models, with theoretical validation and practical applications.

Contribution

It generalizes classical probability metrics to capacities, establishing conditions for validity, and introduces Maximum Mean Imprecision as a new epistemic uncertainty measure validated through experiments.

Findings

IIPM serves as a valid metric and metrizes weak convergence of capacities.

IIPM enables comparison across different IP models and quantifies epistemic uncertainty.

MMI outperforms existing EU measures in selective classification experiments.

Abstract

Quantifying differences between probability distributions is fundamental to statistics and machine learning, primarily for comparing statistical uncertainty. In contrast, epistemic uncertainty -- due to incomplete knowledge -- requires richer representations than those offered by classical probability. Imprecise probability (IP) theory offers such models, capturing ambiguity and partial belief. This has driven growing interest in imprecise probabilistic machine learning (IPML), where inference and decision-making rely on broader uncertainty models -- highlighting the need for metrics beyond classical probability. This work introduces the integral imprecise probability metric framework, a Choquet integral-based generalisation of classical integral probability metrics to the setting of capacities -- a broad class of IP models encompassing many existing ones, including lower probabilities, probability intervals, belief functions, and more. Theoretically, we establish conditions under which IIPM serves as a valid metric and metrises a form of weak convergence of capacities. Practically, IIPM not only enables comparison across different IP models but also supports the quantification of epistemic uncertainty~(EU) within a single IP model. In particular, by comparing an IP model with its conjugate, IIPM gives rise to a new class of epistemic uncertainty measures -- Maximum Mean Imprecision -- which satisfy key axiomatic properties proposed in the uncertainty quantification literature. We validate MMI through selective classification experiments, demonstrating strong empirical performance against established EU measures, and outperforming them when classical methods struggle to scale to a large number of classes. Our work advances both theory and practice in Imprecise Probabilistic Machine Learning, offering a principled framework for comparing and quantifying epistemic uncertainty under imprecision.