Unified Uncertainty Calibration

TL;DR

Unified Uncertainty Calibration (U2C) is a comprehensive framework that combines different uncertainty sources to improve AI classifier calibration, enabling better abstention decisions and outperforming traditional reject-or-classify methods on ImageNet benchmarks.

Contribution

The paper introduces U2C, a novel framework that unifies aleatoric and epistemic uncertainties, addressing communication issues, miscalibration, and misspecification in uncertainty estimates.

Findings

U2C outperforms reject-or-classify methods on ImageNet benchmarks.

Provides a theoretical analysis of uncertainty estimation.

Improves calibration and decision-making in AI classifiers.

Abstract

To build robust, fair, and safe AI systems, we would like our classifiers to say ``I don't know'' when facing test examples that are difficult or fall outside of the training classes.The ubiquitous strategy to predict under uncertainty is the simplistic \emph{reject-or-classify} rule: abstain from prediction if epistemic uncertainty is high, classify otherwise.Unfortunately, this recipe does not allow different sources of uncertainty to communicate with each other, produces miscalibrated predictions, and it does not allow to correct for misspecifications in our uncertainty estimates. To address these three issues, we introduce \emph{unified uncertainty calibration (U2C)}, a holistic framework to combine aleatoric and epistemic uncertainties. U2C enables a clean learning-theoretical analysis of uncertainty estimation, and outperforms reject-or-classify across a variety of ImageNet benchmarks. Our code is available at: https://github.com/facebookresearch/UnifiedUncertaintyCalibration