Improved Depth Estimation of Bayesian Neural Networks

TL;DR

This paper introduces a novel approach for estimating the depth of Bayesian neural networks using a discrete truncated normal distribution, enhancing accuracy and reducing variance in depth estimates.

Contribution

It presents a new method for depth estimation in Bayesian neural networks that independently learns mean and variance, improving upon previous techniques.

Findings

Improved test accuracy on spiral data set

Reduced variance in posterior depth estimates

Effective variational inference for depth modeling

Abstract

This paper proposes improvements over earlier work by Nazareth and Blei (2022) for estimating the depth of Bayesian neural networks. Here, we propose a discrete truncated normal distribution over the network depth to independently learn its mean and variance. Posterior distributions are inferred by minimizing the variational free energy, which balances the model complexity and accuracy. Our method improves test accuracy on the spiral data set and reduces the variance in posterior depth estimates.