SAE: Sequential Anchored Ensembles

TL;DR

Sequential Anchored Ensembles (SAE) offer a computationally efficient method for approximating Bayesian posteriors in neural networks by training ensemble members sequentially with correlated losses, outperforming traditional anchored ensembles in some benchmarks.

Contribution

SAE introduces a sequential training approach with correlated losses, reducing computational costs while maintaining or improving approximation quality compared to anchored ensembles.

Findings

SAE outperforms anchored ensembles on some benchmarks.

SAE achieves 2nd and 3rd place in NeurIPS 2021 competition.

SAE provides a lightweight alternative for Bayesian neural network approximation.

Abstract

Computing the Bayesian posterior of a neural network is a challenging task due to the high-dimensionality of the parameter space. Anchored ensembles approximate the posterior by training an ensemble of neural networks on anchored losses designed for the optima to follow the Bayesian posterior. Training an ensemble, however, becomes computationally expensive as its number of members grows since the full training procedure is repeated for each member. In this note, we present Sequential Anchored Ensembles (SAE), a lightweight alternative to anchored ensembles. Instead of training each member of the ensemble from scratch, the members are trained sequentially on losses sampled with high auto-correlation, hence enabling fast convergence of the neural networks and efficient approximation of the Bayesian posterior. SAE outperform anchored ensembles, for a given computational budget, on some benchmarks while showing comparable performance on the others and achieved 2nd and 3rd place in the light and extended tracks of the NeurIPS 2021 Approximate Inference in Bayesian Deep Learning competition.