Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors

TL;DR

This paper introduces a structured variational learning approach for Bayesian neural networks with horseshoe priors, enhancing model compactness and predictive accuracy, particularly in small-sample scenarios like reinforcement learning.

Contribution

It proposes novel modeling and inference techniques that improve the sparsity and efficiency of Bayesian neural networks using horseshoe priors.

Findings

Enhanced model compactness without sacrificing accuracy

Improved performance in small-sample and reinforcement learning settings

Consistent inference improvements over previous methods

Abstract

Bayesian Neural Networks (BNNs) have recently received increasing attention for their ability to provide well-calibrated posterior uncertainties. However, model selection---even choosing the number of nodes---remains an open question. Recent work has proposed the use of a horseshoe prior over node pre-activations of a Bayesian neural network, which effectively turns off nodes that do not help explain the data. In this work, we propose several modeling and inference advances that consistently improve the compactness of the model learned while maintaining predictive performance, especially in smaller-sample settings including reinforcement learning.