Variational Bayesian Bow tie Neural Networks with Shrinkage

TL;DR

This paper introduces a Bayesian neural network approach with sparsity-promoting priors and Polya-Gamma augmentation, enabling robust, scalable uncertainty estimation without strong independence assumptions.

Contribution

It proposes a novel variational inference method for Bayesian neural networks that relaxes independence assumptions and enhances robustness through sparsity priors.

Findings

Improved robustness to architectural choices

Fast approximate inference algorithm

Enhanced uncertainty estimation capabilities

Abstract

Despite the dominant role of deep models in machine learning, limitations persist, including overconfident predictions, susceptibility to adversarial attacks, and underestimation of variability in predictions. The Bayesian paradigm provides a natural framework to overcome such issues and has become the gold standard for uncertainty estimation with deep models, also providing improved accuracy and a framework for tuning critical hyperparameters. However, exact Bayesian inference is challenging, typically involving variational algorithms that impose strong independence and distributional assumptions. Moreover, existing methods are sensitive to the architectural choice of the network. We address these issues by focusing on a stochastic relaxation of the standard feed-forward rectified neural network and using sparsity-promoting priors on the weights of the neural network for increased robustness to architectural design. Thanks to Polya-Gamma data augmentation tricks, which render a conditionally linear and Gaussian model, we derive a fast, approximate variational inference algorithm that avoids distributional assumptions and independence across layers. Suitable strategies to further improve scalability and account for multimodality are considered.