Projected BNNs: Avoiding weight-space pathologies by learning latent representations of neural network weights

TL;DR

This paper introduces a variational inference framework for Bayesian neural networks that learns low-dimensional latent representations of weights, improving uncertainty quantification and generalization across diverse datasets.

Contribution

It proposes a novel approach to encode complex high-dimensional weight distributions in a low-dimensional latent space for efficient inference.

Findings

Enhanced uncertainty estimation over traditional methods

Improved model generalization demonstrated on multiple datasets

Efficient inference achieved through low-dimensional representations

Abstract

As machine learning systems get widely adopted for high-stake decisions, quantifying uncertainty over predictions becomes crucial. While modern neural networks are making remarkable gains in terms of predictive accuracy, characterizing uncertainty over the parameters of these models is challenging because of the high dimensionality and complex correlations of the network parameter space. This paper introduces a novel variational inference framework for Bayesian neural networks that (1) encodes complex distributions in high-dimensional parameter space with representations in a low-dimensional latent space, and (2) performs inference efficiently on the low-dimensional representations. Across a large array of synthetic and real-world datasets, we show that our method improves uncertainty characterization and model generalization when compared with methods that work directly in the parameter space.