Spike-and-slab shrinkage priors for structurally sparse Bayesian neural networks

TL;DR

This paper introduces structurally sparse Bayesian neural networks using Spike-and-Slab priors to achieve model compression and efficiency, with theoretical guarantees and competitive empirical results.

Contribution

It proposes novel variational inference methods for structured sparsity in Bayesian neural networks using Spike-and-Slab priors, with theoretical contraction rate analysis.

Findings

Competitive prediction accuracy compared to baseline models

Effective model compression and reduced inference latency

Theoretical contraction rates established for the proposed models

Abstract

Network complexity and computational efficiency have become increasingly significant aspects of deep learning. Sparse deep learning addresses these challenges by recovering a sparse representation of the underlying target function by reducing heavily over-parameterized deep neural networks. Specifically, deep neural architectures compressed via structured sparsity (e.g. node sparsity) provide low latency inference, higher data throughput, and reduced energy consumption. In this paper, we explore two well-established shrinkage techniques, Lasso and Horseshoe, for model compression in Bayesian neural networks. To this end, we propose structurally sparse Bayesian neural networks which systematically prune excessive nodes with (i) Spike-and-Slab Group Lasso (SS-GL), and (ii) Spike-and-Slab Group Horseshoe (SS-GHS) priors, and develop computationally tractable variational inference including continuous relaxation of Bernoulli variables. We establish the contraction rates of the variational posterior of our proposed models as a function of the network topology, layer-wise node cardinalities, and bounds on the network weights. We empirically demonstrate the competitive performance of our models compared to the baseline models in prediction accuracy, model compression, and inference latency.