Walking on the Fiber: A Simple Geometric Approximation for Bayesian Neural Networks

TL;DR

This paper introduces a novel, efficient method for sampling from the posterior distribution of Bayesian Neural Networks by leveraging low-dimensional structures and learned deformations, improving scalability and approximation quality.

Contribution

It proposes a simple, scalable approach for posterior sampling in over-parameterized networks using geometric deformations of the parameter space.

Findings

Achieves competitive posterior approximations

Improves scalability over existing methods

Provides a practical alternative for Bayesian deep learning

Abstract

Bayesian Neural Networks provide a principled framework for uncertainty quantification by modeling the posterior distribution of network parameters. However, exact posterior inference is computationally intractable, and widely used approximations like the Laplace method struggle with scalability and posterior accuracy in modern deep networks. In this work, we revisit sampling techniques for posterior exploration, proposing a simple variation tailored to efficiently sample from the posterior in over-parameterized networks by leveraging the low-dimensional structure of loss minima. Building on this, we introduce a model that learns a deformation of the parameter space, enabling rapid posterior sampling without requiring iterative methods. Empirical results demonstrate that our approach achieves competitive posterior approximations with improved scalability compared to recent refinement techniques. These contributions provide a practical alternative for Bayesian inference in deep learning.