Overpruning in Variational Bayesian Neural Networks

TL;DR

This paper investigates the counter-intuitive phenomenon where more expressive variational approximations in Bayesian neural networks can lead to worse predictions, identifying over-pruning as a key cause and providing a theoretical explanation.

Contribution

It identifies variational over-pruning as a cause of performance degradation and offers a theoretical framework to understand this phenomenon in Bayesian neural networks.

Findings

More expressive variational families can worsen predictions due to over-pruning.

The paper provides a theoretical explanation for over-pruning in variational Bayesian neural networks.

Insights help guide the design of better variational approximations.

Abstract

The motivations for using variational inference (VI) in neural networks differ significantly from those in latent variable models. This has a counter-intuitive consequence; more expressive variational approximations can provide significantly worse predictions as compared to those with less expressive families. In this work we make two contributions. First, we identify a cause of this performance gap, variational over-pruning. Second, we introduce a theoretically grounded explanation for this phenomenon. Our perspective sheds light on several related published results and provides intuition into the design of effective variational approximations of neural networks.