Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty

TL;DR

This paper introduces a scalable variational inference method for Bayesian neural networks that jointly models structural and parameter uncertainty, leading to sparser models with competitive accuracy.

Contribution

It presents a novel approach combining model and parameter uncertainty in BNNs using variational inference with reparametrized marginal inclusion probabilities.

Findings

Achieves comparable accuracy to existing models

Produces sparser Bayesian neural networks

Demonstrates effectiveness on benchmark datasets

Abstract

Bayesian neural networks (BNNs) have recently regained a significant amount of attention in the deep learning community due to the development of scalable approximate Bayesian inference techniques. There are several advantages of using a Bayesian approach: Parameter and prediction uncertainties become easily available, facilitating rigorous statistical analysis. Furthermore, prior knowledge can be incorporated. However, so far, there have been no scalable techniques capable of combining both structural and parameter uncertainty. In this paper, we apply the concept of model uncertainty as a framework for structural learning in BNNs and hence make inference in the joint space of structures/models and parameters. Moreover, we suggest an adaptation of a scalable variational inference approach with reparametrization of marginal inclusion probabilities to incorporate the model space constraints. Experimental results on a range of benchmark datasets show that we obtain comparable accuracy results with the competing models, but based on methods that are much more sparse than ordinary BNNs.