Sampling-free Variational Inference for Neural Networks with Multiplicative Activation Noise

TL;DR

This paper introduces a more efficient sampling-free variational inference method for Bayesian neural networks using multiplicative Gaussian activation noise, achieving competitive results in regression and large-scale image classification.

Contribution

It proposes a novel parameterization of the posterior that combines sampling-free inference with parameter efficiency, improving scalability and performance.

Findings

Competitive results on standard regression tasks

Effective scaling to large datasets like ImageNet

Reduced parameter overhead compared to previous methods

Abstract

To adopt neural networks in safety critical domains, knowing whether we can trust their predictions is crucial. Bayesian neural networks (BNNs) provide uncertainty estimates by averaging predictions with respect to the posterior weight distribution. Variational inference methods for BNNs approximate the intractable weight posterior with a tractable distribution, yet mostly rely on sampling from the variational distribution during training and inference. Recent sampling-free approaches offer an alternative, but incur a significant parameter overhead. We here propose a more efficient parameterization of the posterior approximation for sampling-free variational inference that relies on the distribution induced by multiplicative Gaussian activation noise. This allows us to combine parameter efficiency with the benefits of sampling-free variational inference. Our approach yields competitive results for standard regression problems and scales well to large-scale image classification tasks including ImageNet.