Bayesian Lottery Ticket Hypothesis

TL;DR

This paper investigates the existence and properties of sparse subnetworks, called Bayesian lottery tickets, within Bayesian neural networks, showing they can match or surpass dense network performance with implications for efficient training.

Contribution

It extends the Lottery Ticket Hypothesis to Bayesian neural networks, analyzing characteristics of sparse subnetworks and proposing a transplantation method linking BNNs with deterministic lottery tickets.

Findings

LTH holds in BNNs with sparse subnetworks.

Winning tickets are consistent across model sizes.

Pruning should prioritize magnitude and standard deviation.

Abstract

Bayesian neural networks (BNNs) are a useful tool for uncertainty quantification, but require substantially more computational resources than conventional neural networks. For non-Bayesian networks, the Lottery Ticket Hypothesis (LTH) posits the existence of sparse subnetworks that can train to the same or even surpassing accuracy as the original dense network. Such sparse networks can lower the demand for computational resources at inference, and during training. The existence of the LTH and corresponding sparse subnetworks in BNNs could motivate the development of sparse training algorithms and provide valuable insights into the underlying training process. Towards this end, we translate the LTH experiments to a Bayesian setting using common computer vision models. We investigate the defining characteristics of Bayesian lottery tickets, and extend our study towards a transplantation method connecting BNNs with deterministic Lottery Tickets. We generally find that the LTH holds in BNNs, and winning tickets of matching and surpassing accuracy are present independent of model size, with degradation at very high sparsities. However, the pruning strategy should rely primarily on magnitude, secondly on standard deviation. Furthermore, our results demonstrate that models rely on mask structure and weight initialization to varying degrees.