'In-Between' Uncertainty in Bayesian Neural Networks

TL;DR

This paper identifies a limitation of mean-field variational inference in Bayesian neural networks, showing it fails to provide calibrated uncertainty estimates between observed data regions, and proposes the linearised Laplace approximation as a better alternative for small networks.

Contribution

It highlights the limitations of MFVI in capturing 'in-between' uncertainty and demonstrates the effectiveness of the linearised Laplace approximation for small architectures.

Findings

MFVI fails to calibrate uncertainty between separated data regions.

Linearised Laplace approximation improves 'in-between' uncertainty estimates.

Better uncertainty calibration enhances out-of-distribution robustness.

Abstract

We describe a limitation in the expressiveness of the predictive uncertainty estimate given by mean-field variational inference (MFVI), a popular approximate inference method for Bayesian neural networks. In particular, MFVI fails to give calibrated uncertainty estimates in between separated regions of observations. This can lead to catastrophically overconfident predictions when testing on out-of-distribution data. Avoiding such overconfidence is critical for active learning, Bayesian optimisation and out-of-distribution robustness. We instead find that a classical technique, the linearised Laplace approximation, can handle 'in-between' uncertainty much better for small network architectures.