Masked Bayesian Neural Networks : Computation and Optimality

TL;DR

This paper introduces a sparse Bayesian neural network that adaptively simplifies deep neural networks by turning off nodes, achieving optimality and competitive performance with smaller architectures.

Contribution

It proposes a novel masking-based Bayesian neural network with a specially designed prior for optimality and an efficient MCMC algorithm for training.

Findings

Performs well on benchmark datasets

Discovers compact DNN architectures

Maintains prediction accuracy and uncertainty quantification

Abstract

As data size and computing power increase, the architectures of deep neural networks (DNNs) have been getting more complex and huge, and thus there is a growing need to simplify such complex and huge DNNs. In this paper, we propose a novel sparse Bayesian neural network (BNN) which searches a good DNN with an appropriate complexity. We employ the masking variables at each node which can turn off some nodes according to the posterior distribution to yield a nodewise sparse DNN. We devise a prior distribution such that the posterior distribution has theoretical optimalities (i.e. minimax optimality and adaptiveness), and develop an efficient MCMC algorithm. By analyzing several benchmark datasets, we illustrate that the proposed BNN performs well compared to other existing methods in the sense that it discovers well condensed DNN architectures with similar prediction accuracy and uncertainty quantification compared to large DNNs.