Bayesian posterior approximation with stochastic ensembles

TL;DR

This paper proposes stochastic neural network ensembles that combine dropout and deep ensembles to better approximate Bayesian posteriors, demonstrating improved accuracy over existing methods on classification tasks.

Contribution

It introduces stochastic ensembles as a new variational inference approach for Bayesian approximation, integrating dropout techniques with ensemble methods.

Findings

Stochastic ensembles outperform traditional methods in posterior estimation accuracy.

They are effective on both toy problems and real-world image classification.

The approach provides a flexible framework for Bayesian neural network inference.

Abstract

We introduce ensembles of stochastic neural networks to approximate the Bayesian posterior, combining stochastic methods such as dropout with deep ensembles. The stochastic ensembles are formulated as families of distributions and trained to approximate the Bayesian posterior with variational inference. We implement stochastic ensembles based on Monte Carlo dropout, DropConnect and a novel non-parametric version of dropout and evaluate them on a toy problem and CIFAR image classification. For both tasks, we test the quality of the posteriors directly against Hamiltonian Monte Carlo simulations. Our results show that stochastic ensembles provide more accurate posterior estimates than other popular baselines for Bayesian inference.