On the Convergence of Locally Adaptive and Scalable Diffusion-Based Sampling Methods for Deep Bayesian Neural Network Posteriors

TL;DR

This paper investigates the convergence properties of adaptive diffusion-based sampling methods for Bayesian neural networks, revealing potential biases even with small step sizes and full batch data, impacting uncertainty quantification.

Contribution

It critically analyzes existing adaptive sampling algorithms, demonstrating their potential bias and limitations in accurately sampling from neural network posteriors.

Findings

Existing methods can have substantial bias in the sampled distribution.

Bias persists even with vanishing step sizes and full batch data.

Challenges in achieving reliable uncertainty quantification in deep learning.

Abstract

Achieving robust uncertainty quantification for deep neural networks represents an important requirement in many real-world applications of deep learning such as medical imaging where it is necessary to assess the reliability of a neural network's prediction. Bayesian neural networks are a promising approach for modeling uncertainties in deep neural networks. Unfortunately, generating samples from the posterior distribution of neural networks is a major challenge. One significant advance in that direction would be the incorporation of adaptive step sizes, similar to modern neural network optimizers, into Monte Carlo Markov chain sampling algorithms without significantly increasing computational demand. Over the past years, several papers have introduced sampling algorithms with claims that they achieve this property. However, do they indeed converge to the correct distribution? In this paper, we demonstrate that these methods can have a substantial bias in the distribution they sample, even in the limit of vanishing step sizes and at full batch size.