Deep Ensembles Secretly Perform Empirical Bayes

TL;DR

This paper reveals that deep ensembles perform exact Bayesian averaging with a data-dependent prior, providing a theoretical justification for their empirical success and bridging the gap between ensembles and Bayesian neural networks.

Contribution

It demonstrates that deep ensembles are equivalent to empirical Bayes procedures with a learned data-dependent prior, offering a new theoretical understanding.

Findings

Deep ensembles perform exact Bayesian averaging.

The learned prior is a mixture of point masses.

This perspective explains the strong empirical performance of ensembles.

Abstract

Quantifying uncertainty in neural networks is a highly relevant problem which is essential to many applications. The two predominant paradigms to tackle this task are Bayesian neural networks (BNNs) and deep ensembles. Despite some similarities between these two approaches, they are typically surmised to lack a formal connection and are thus understood as fundamentally different. BNNs are often touted as more principled due to their reliance on the Bayesian paradigm, whereas ensembles are perceived as more ad-hoc; yet, deep ensembles tend to empirically outperform BNNs, with no satisfying explanation as to why this is the case. In this work we bridge this gap by showing that deep ensembles perform exact Bayesian averaging with a posterior obtained with an implicitly learned data-dependent prior. In other words deep ensembles are Bayesian, or more specifically, they implement an empirical Bayes procedure wherein the prior is learned from the data. This perspective offers two main benefits: (i) it theoretically justifies deep ensembles and thus provides an explanation for their strong empirical performance; and (ii) inspection of the learned prior reveals it is given by a mixture of point masses -- the use of such a strong prior helps elucidate observed phenomena about ensembles. Overall, our work delivers a newfound understanding of deep ensembles which is not only of interest in it of itself, but which is also likely to generate future insights that drive empirical improvements for these models.