Tubular Riemannian Laplace Approximations for Bayesian Neural Networks

TL;DR

The paper introduces the Tubular Riemannian Laplace (TRL) approximation, a scalable Bayesian inference method for neural networks that models the posterior as a probabilistic tube, improving calibration and efficiency over existing methods.

Contribution

It proposes TRL, a novel Riemannian Laplace approximation that explicitly models posterior geometry using a Fisher/Gauss-Newton metric, enhancing scalability and accuracy.

Findings

TRL achieves excellent calibration on ResNet-18.

TRL matches or exceeds Deep Ensembles in reliability.

TRL requires only 1/5 of the training cost.

Abstract

Laplace approximations are among the simplest and most practical methods for approximate Bayesian inference in neural networks, yet their Euclidean formulation struggles with the highly anisotropic, curved loss surfaces and large symmetry groups that characterize modern deep models. Recent work has proposed Riemannian and geometric Gaussian approximations to adapt to this structure. Building on these ideas, we introduce the Tubular Riemannian Laplace (TRL) approximation. TRL explicitly models the posterior as a probabilistic tube that follows a low-loss valley induced by functional symmetries, using a Fisher/Gauss-Newton metric to separate prior-dominated tangential uncertainty from data-dominated transverse uncertainty. We interpret TRL as a scalable reparametrised Gaussian approximation that utilizes implicit curvature estimates to operate in high-dimensional parameter spaces. Our empirical evaluation on ResNet-18 (CIFAR-10 and CIFAR-100) demonstrates that TRL achieves excellent calibration, matching or exceeding the reliability of Deep Ensembles (in terms of ECE) while requiring only a fraction (1/5) of the training cost. TRL effectively bridges the gap between single-model efficiency and ensemble-grade reliability.