Isotropic SGD: a Practical Approach to Bayesian Posterior Sampling

TL;DR

This paper introduces a practical method for Bayesian posterior sampling using isotropic stochastic gradient noise, improving upon existing algorithms by requiring weaker assumptions and maintaining competitive performance.

Contribution

It proposes a novel approach to make SG noise isotropic with a fixed learning rate, simplifying and enhancing the practicality of SG-MCMC algorithms.

Findings

Competitive performance with state-of-the-art SG-MCMC methods

Requires weaker assumptions than existing algorithms

More practical to implement and tune

Abstract

In this work we define a unified mathematical framework to deepen our understanding of the role of stochastic gradient (SG) noise on the behavior of Markov chain Monte Carlo sampling (SGMCMC) algorithms. Our formulation unlocks the design of a novel, practical approach to posterior sampling, which makes the SG noise isotropic using a fixed learning rate that we determine analytically, and that requires weaker assumptions than existing algorithms. In contrast, the common traits of existing \sgmcmc algorithms is to approximate the isotropy condition either by drowning the gradients in additive noise (annealing the learning rate) or by making restrictive assumptions on the \sg noise covariance and the geometry of the loss landscape. Extensive experimental validations indicate that our proposal is competitive with the state-of-the-art on \sgmcmc, while being much more practical to use.