Analog Bayesian neural networks are insensitive to the shape of the weight distribution

TL;DR

This paper shows that in analog Bayesian neural networks trained with mean field variational inference, the predictive distribution is insensitive to the shape of the weight distribution, simplifying hardware design considerations.

Contribution

The paper introduces a method for training Bayesian neural networks using real device noise as the variational distribution and demonstrates the insensitivity of predictive distributions to the shape of the weight distribution.

Findings

Predictive distributions converge regardless of the variational distribution shape.

Analog device noise shape does not affect BNN predictive performance.

Method enables hardware implementation without precise noise shape control.

Abstract

Recent work has demonstrated that Bayesian neural networks (BNN's) trained with mean field variational inference (MFVI) can be implemented in analog hardware, promising orders of magnitude energy savings compared to the standard digital implementations. However, while Gaussians are typically used as the variational distribution in MFVI, it is difficult to precisely control the shape of the noise distributions produced by sampling analog devices. This paper introduces a method for MFVI training using real device noise as the variational distribution. Furthermore, we demonstrate empirically that the predictive distributions from BNN's with the same weight means and variances converge to the same distribution, regardless of the shape of the variational distribution. This result suggests that analog device designers do not need to consider the shape of the device noise distribution when hardware-implementing BNNs performing MFVI.