Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning

TL;DR

This paper advocates for the adoption of sampling-based inference in Bayesian neural networks, highlighting its current parity with optimization methods and its potential to improve uncertainty quantification and prediction.

Contribution

It argues that sampling-based inference has achieved computational parity with optimization methods and should be prioritized for advancing Bayesian neural networks.

Findings

Sampling-based inference is now computationally comparable to optimization methods.

SAI can provide superior prediction performance through model averaging.

Addressing exploration and distillation challenges can unlock SAI's full potential.

Abstract

The practical adoption of sampling-based inference (SAI) in Bayesian neural networks (BNNs) remains limited, partly due to persistent misconceptions about the feasibility and efficiency of sampling. This position paper argues that SAI has achieved computational parity with optimization-based methods and is at the verge of superseding such methods for effective and efficient inference in BNNs. This development should be in the interest of the whole community, promoting BNNs as a principled paradigm with its long-standing yet unfulfilled promise of providing principled uncertainty quantification for neural networks. SAI can even do more -- yielding superior prediction performance through model averaging, serving as the foundation for a plethora of possible downstream tasks, and providing crucial insights into the landscape of BNNs. In order to make such a change happen and unfold the potential of sampling, overcoming current misconceptions is a necessary first step. The next step is to realign research efforts toward addressing remaining challenges in SAI. In particular, the community must focus on two core problems: sufficient exploration of the posterior landscape and high-fidelity distillation of posterior samples for efficient downstream inference. By addressing conceptual and practical obstacles, we can unlock the full potential of SAI and establish it as a central tool in Bayesian deep learning.