Data Subsampling for Bayesian Neural Networks

TL;DR

This paper introduces Penalty Bayesian Neural Networks (PBNNs), a scalable MCMC-based method that uses data subsampling with a penalty term to improve Bayesian neural network inference on large datasets.

Contribution

The paper proposes PBNNs, a novel subsampling algorithm for Bayesian neural networks that reduces bias and integrates easily with existing MCMC frameworks.

Findings

PBNN achieves good predictive performance with small mini-batches.

PBNN reduces predictive overconfidence by varying mini-batch size.

PBNN is effective on synthetic data and MNIST dataset.

Abstract

Markov Chain Monte Carlo (MCMC) algorithms do not scale well for large datasets leading to difficulties in Neural Network posterior sampling. In this paper, we propose Penalty Bayesian Neural Networks - PBNNs, as a new algorithm that allows the evaluation of the likelihood using subsampled batch data (mini-batches) in a Bayesian inference context towards addressing scalability. PBNN avoids the biases inherent in other naive subsampling techniques by incorporating a penalty term as part of a generalization of the Metropolis Hastings algorithm. We show that it is straightforward to integrate PBNN with existing MCMC frameworks, as the variance of the loss function merely reduces the acceptance probability. By comparing with alternative sampling strategies on both synthetic data and the MNIST dataset, we demonstrate that PBNN achieves good predictive performance even for small mini-batch sizes of data. We show that PBNN provides a novel approach for calibrating the predictive distribution by varying the mini-batch size, significantly reducing predictive overconfidence.