Structured Partial Stochasticity in Bayesian Neural Networks

TL;DR

This paper introduces a structured approach to select deterministic weights in Bayesian neural networks, reducing redundant modes and improving the efficiency and performance of approximate inference methods.

Contribution

It proposes a novel structured method to eliminate neuron permutation symmetries, simplifying the posterior distribution in Bayesian neural networks.

Findings

Simplified posterior improves inference performance

Reduces redundant modes in Bayesian neural network posteriors

Enhances efficiency of approximate inference methods

Abstract

Bayesian neural network posterior distributions have a great number of modes that correspond to the same network function. The abundance of such modes can make it difficult for approximate inference methods to do their job. Recent work has demonstrated the benefits of partial stochasticity for approximate inference in Bayesian neural networks; inference can be less costly and performance can sometimes be improved. I propose a structured way to select the deterministic subset of weights that removes neuron permutation symmetries, and therefore the corresponding redundant posterior modes. With a drastically simplified posterior distribution, the performance of existing approximate inference schemes is found to be greatly improved.