Deep Ensembles for Epistemic Uncertainty: A Frequentist Perspective

TL;DR

This paper provides a theoretical analysis of deep ensembles for epistemic uncertainty, demonstrating they capture training stochasticity and explaining their empirical success from a frequentist perspective.

Contribution

It introduces a bootstrap-based estimator for epistemic uncertainty and shows deep ensembles approximate this estimator by capturing training stochasticity.

Findings

Deep ensembles primarily capture training stochasticity.

The bootstrap estimator for epistemic uncertainty is asymptotically correct.

Empirical results confirm the dominance of training stochasticity in epistemic uncertainty.

Abstract

Decomposing prediction uncertainty into aleatoric (irreducible) and epistemic (reducible) components is critical for the reliable deployment of machine learning systems. While the mutual information between the response variable and model parameters is a principled measure for epistemic uncertainty, it requires access to the parameter posterior, which is computationally challenging to approximate. Consequently, practitioners often rely on probabilistic predictions from deep ensembles to quantify uncertainty, which have demonstrated strong empirical performance. However, a theoretical understanding of their success from a frequentist perspective remains limited. We address this gap by first considering a bootstrap-based estimator for epistemic uncertainty, which we prove is asymptotically correct. Next, we connect deep ensembles to the bootstrap estimator by decomposing it into data variability and training stochasticity; specifically, we show that deep ensembles capture the training stochasticity component. Through empirical studies, we show that this stochasticity component constitutes the majority of epistemic uncertainty, thereby explaining the effectiveness of deep ensembles.