Quality of Uncertainty Quantification for Bayesian Neural Network Inference

TL;DR

This paper empirically evaluates the quality of uncertainty estimates in Bayesian Neural Networks across various inference methods, revealing limitations of common metrics and the disconnect between structural inference improvements and posterior quality.

Contribution

It provides a comprehensive empirical comparison of 10 inference methods for BNNs, highlighting issues with standard evaluation metrics and the effectiveness of structural inference innovations.

Findings

Common metrics like test log-likelihood can be misleading

Structural inference improvements do not always yield better posterior approximations

Many inference methods produce similar uncertainty quality despite different approaches

Abstract

Bayesian Neural Networks (BNNs) place priors over the parameters in a neural network. Inference in BNNs, however, is difficult; all inference methods for BNNs are approximate. In this work, we empirically compare the quality of predictive uncertainty estimates for 10 common inference methods on both regression and classification tasks. Our experiments demonstrate that commonly used metrics (e.g. test log-likelihood) can be misleading. Our experiments also indicate that inference innovations designed to capture structure in the posterior do not necessarily produce high quality posterior approximations.