Credal Ensemble Distillation for Uncertainty Quantification

TL;DR

This paper introduces credal ensemble distillation (CED), a method that compresses deep ensembles into a single model predicting probability intervals, improving uncertainty quantification and reducing inference costs.

Contribution

The paper presents a novel framework, CED, that distills deep ensembles into a single model with credal sets for better uncertainty estimation and efficiency.

Findings

CED achieves comparable or superior uncertainty estimation to deep ensembles.

CED significantly reduces inference costs compared to traditional deep ensembles.

Empirical results on out-of-distribution detection benchmarks validate CED's effectiveness.

Abstract

Deep ensembles (DE) have emerged as a powerful approach for quantifying predictive uncertainty and distinguishing its aleatoric and epistemic components, thereby enhancing model robustness and reliability. However, their high computational and memory costs during inference pose significant challenges for wide practical deployment. To overcome this issue, we propose credal ensemble distillation (CED), a novel framework that compresses a DE into a single model, CREDIT, for classification tasks. Instead of a single softmax probability distribution, CREDIT predicts class-wise probability intervals that define a credal set, a convex set of probability distributions, for uncertainty quantification. Empirical results on out-of-distribution detection benchmarks demonstrate that CED achieves superior or comparable uncertainty estimation compared to several existing baselines, while substantially reducing inference overhead compared to DE.