Unified Uncertainty Calibration
Kamalika Chaudhuri, David Lopez-Paz

TL;DR
Unified Uncertainty Calibration (U2C) is a comprehensive framework that combines different uncertainty sources to improve AI classifier calibration, enabling better abstention decisions and outperforming traditional reject-or-classify methods on ImageNet benchmarks.
Contribution
The paper introduces U2C, a novel framework that unifies aleatoric and epistemic uncertainties, addressing communication issues, miscalibration, and misspecification in uncertainty estimates.
Findings
U2C outperforms reject-or-classify methods on ImageNet benchmarks.
Provides a theoretical analysis of uncertainty estimation.
Improves calibration and decision-making in AI classifiers.
Abstract
To build robust, fair, and safe AI systems, we would like our classifiers to say ``I don't know'' when facing test examples that are difficult or fall outside of the training classes.The ubiquitous strategy to predict under uncertainty is the simplistic \emph{reject-or-classify} rule: abstain from prediction if epistemic uncertainty is high, classify otherwise.Unfortunately, this recipe does not allow different sources of uncertainty to communicate with each other, produces miscalibrated predictions, and it does not allow to correct for misspecifications in our uncertainty estimates. To address these three issues, we introduce \emph{unified uncertainty calibration (U2C)}, a holistic framework to combine aleatoric and epistemic uncertainties. U2C enables a clean learning-theoretical analysis of uncertainty estimation, and outperforms reject-or-classify across a variety of ImageNet…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsFault Detection and Control Systems
