Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning

TL;DR

Radial Bayesian Neural Networks introduce a scalable variational posterior with full support over weight-space, overcoming limitations of discrete support methods and excelling in large-scale Bayesian deep learning applications.

Contribution

The paper presents Radial BNNs, a novel variational approach with full support, addressing the 'soap-bubble' pathology and demonstrating superior performance in real-world and continual learning tasks.

Findings

Radial BNNs outperform MC dropout in medical applications.

They are robust to hyperparameters and require less tuning.

Radial BNNs effectively handle large-scale models with full weight-space support.

Abstract

We propose Radial Bayesian Neural Networks (BNNs): a variational approximate posterior for BNNs which scales well to large models while maintaining a distribution over weight-space with full support. Other scalable Bayesian deep learning methods, like MC dropout or deep ensembles, have discrete support-they assign zero probability to almost all of the weight-space. Unlike these discrete support methods, Radial BNNs' full support makes them suitable for use as a prior for sequential inference. In addition, they solve the conceptual challenges with the a priori implausibility of weight distributions with discrete support. The Radial BNN is motivated by avoiding a sampling problem in 'mean-field' variational inference (MFVI) caused by the so-called 'soap-bubble' pathology of multivariate Gaussians. We show that, unlike MFVI, Radial BNNs are robust to hyperparameters and can be efficiently applied to a challenging real-world medical application without needing ad-hoc tweaks and intensive tuning. In fact, in this setting Radial BNNs out-perform discrete-support methods like MC dropout. Lastly, by using Radial BNNs as a theoretically principled, robust alternative to MFVI we make significant strides in a Bayesian continual learning evaluation.