Approximate blocked Gibbs sampling for Bayesian neural networks

TL;DR

This paper introduces an approximate blocked Gibbs sampling method for Bayesian neural networks that improves scalability and uncertainty quantification by partitioning parameters and managing acceptance rates.

Contribution

It proposes a novel minibatch MCMC sampling scheme using blocked Gibbs sampling to handle neural network parameters regardless of layer width.

Findings

Sampling subgroups of parameters improves scalability.

Reducing proposal variance alleviates vanishing acceptance rates.

Longer non-convergent chains enhance predictive accuracy and uncertainty quantification.

Abstract

In this work, minibatch MCMC sampling for feedforward neural networks is made more feasible. To this end, it is proposed to sample subgroups of parameters via a blocked Gibbs sampling scheme. By partitioning the parameter space, sampling is possible irrespective of layer width. It is also possible to alleviate vanishing acceptance rates for increasing depth by reducing the proposal variance in deeper layers. Increasing the length of a non-convergent chain increases the predictive accuracy in classification tasks, so avoiding vanishing acceptance rates and consequently enabling longer chain runs have practical benefits. Moreover, non-convergent chain realizations aid in the quantification of predictive uncertainty. An open problem is how to perform minibatch MCMC sampling for feedforward neural networks in the presence of augmented data.