SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batching

TL;DR

This paper develops a theoretical framework for stochastic gradient methods in black-box variational inference, addressing unbounded gradient variance through preconditioning and dynamic batching to ensure convergence.

Contribution

It proves the existence of ELBO solutions and provides convergence guarantees for minibatch projected SGD under the BG condition, with practical numerical validation.

Findings

Existence of ELBO solutions established.

Convergence guarantees for minibatch PSGD with preconditioning.

Numerical results demonstrate practical efficacy.

Abstract

Black-Box Variational Inference (BBVI) typically relies on Stochastic Gradient Descent (SGD) to optimize the Evidence Lower Bound (ELBO). However, the stochastic gradients in BBVI inherently exhibit unbounded variance, violating standard assumptions and instead satisfying the weaker Blum-Gladyshev (BG) condition, where variance grows quadratically with distance from the optimum. In this paper, we bridge the gap between stochastic optimization theory and the practical instances of BBVI. Focusing on the broad elliptic location-scale family of parameterized distributions, we offer two main contributions. First, we prove the existence of an ELBO solution, a foundational property usually assumed a priori in the literature. Second, we establish comprehensive convergence guarantees spanning finite-time and asymptotic regimes for Minibatch Projected SGD (PSGD) equipped with dynamic batching and preconditioning under the BG condition. Our theoretical framework demonstrates that dynamic batching combined with preconditioning systematically enables rigorous guarantees even in complex settings. We illustrate our theoretical findings with numerical results, highlighting the efficacy of our approach for modern inference tasks.