Weight Uncertainty in Neural Networks

TL;DR

This paper presents Bayes by Backprop, an efficient algorithm for learning probabilistic weight distributions in neural networks, improving regularisation, generalisation, and exploration strategies.

Contribution

It introduces a novel backpropagation-compatible method for Bayesian neural networks that models weight uncertainty and enhances performance across various tasks.

Findings

Comparable performance to dropout on MNIST classification

Improved generalisation in non-linear regression

Enhanced exploration in reinforcement learning

Abstract

We introduce a new, efficient, principled and backpropagation-compatible algorithm for learning a probability distribution on the weights of a neural network, called Bayes by Backprop. It regularises the weights by minimising a compression cost, known as the variational free energy or the expected lower bound on the marginal likelihood. We show that this principled kind of regularisation yields comparable performance to dropout on MNIST classification. We then demonstrate how the learnt uncertainty in the weights can be used to improve generalisation in non-linear regression problems, and how this weight uncertainty can be used to drive the exploration-exploitation trade-off in reinforcement learning.