Variational Laplace for Bayesian neural networks

TL;DR

This paper introduces Variational Laplace, a new method for Bayesian neural networks that approximates the likelihood curvature to efficiently estimate the ELBO, improving performance over traditional methods.

Contribution

It presents Variational Laplace, a simple, efficient approximation for Bayesian neural networks that enhances test accuracy and calibration without stochastic sampling.

Findings

Outperforms MAP and standard VI in test performance and calibration

Provides a simple evaluation of the ELBO using local curvature approximation

Highlights the importance of proper convergence criteria in VI benchmarking

Abstract

We develop variational Laplace for Bayesian neural networks (BNNs) which exploits a local approximation of the curvature of the likelihood to estimate the ELBO without the need for stochastic sampling of the neural-network weights. The Variational Laplace objective is simple to evaluate, as it is (in essence) the log-likelihood, plus weight-decay, plus a squared-gradient regularizer. Variational Laplace gave better test performance and expected calibration errors than maximum a-posteriori inference and standard sampling-based variational inference, despite using the same variational approximate posterior. Finally, we emphasise care needed in benchmarking standard VI as there is a risk of stopping before the variance parameters have converged. We show that early-stopping can be avoided by increasing the learning rate for the variance parameters.