Greedy Bayesian Posterior Approximation with Deep Ensembles

TL;DR

This paper introduces a new principled method for constructing neural network ensembles that better approximate the true posterior distribution, improving uncertainty estimation in deep learning, especially for out-of-distribution detection.

Contribution

It proposes a novel greedy ensemble construction method based on minimizing an $f$-divergence, with a new diversity term derived from a combinatorial analysis, enhancing posterior approximation.

Findings

Improved out-of-distribution detection performance.

The method is effective across various architectures and datasets.

Source code is publicly available for reproducibility.

Abstract

Ensembles of independently trained neural networks are a state-of-the-art approach to estimate predictive uncertainty in Deep Learning, and can be interpreted as an approximation of the posterior distribution via a mixture of delta functions. The training of ensembles relies on non-convexity of the loss landscape and random initialization of their individual members, making the resulting posterior approximation uncontrolled. This paper proposes a novel and principled method to tackle this limitation, minimizing an $f$-divergence between the true posterior and a kernel density estimator (KDE) in a function space. We analyze this objective from a combinatorial point of view, and show that it is submodular with respect to mixture components for any $f$. Subsequently, we consider the problem of greedy ensemble construction. From the marginal gain on the negative $f$-divergence, which quantifies an improvement in posterior approximation yielded by adding a new component into the KDE, we derive a novel diversity term for ensemble methods. The performance of our approach is demonstrated on computer vision out-of-distribution detection benchmarks in a range of architectures trained on multiple datasets. The source code of our method is made publicly available at https://github.com/Oulu-IMEDS/greedy_ensembles_training.