Convergence of projected stochastic natural gradient variational inference for various step size and sample or batch size schedules

TL;DR

This paper provides a comprehensive theoretical analysis of the convergence properties of stochastic natural gradient variational inference (NGVI) under various step size and sample size schedules, extending existing results.

Contribution

It establishes new non-asymptotic convergence rates for projected stochastic NGVI with different hyperparameter schedules, including geometric and polynomial rates.

Findings

NGVI converges geometrically to a neighborhood of the optimum with fixed hyperparameters.

Convergence to the exact optimum occurs at rates of O(1/T^ρ) for other hyperparameter schedules.

The results apply when the target distribution is close to the exponential family considered.

Abstract

Stochastic natural gradient variational inference (NGVI) is a popular and efficient algorithm for Bayesian inference. Despite empirical success, the convergence of this method is still not fully understood. In this work, we define and study a projected stochastic NGVI when variational distributions form an exponential family. Stochasticity arises when either gradients are intractable expectations or large sums. We prove new non-asymptotic convergence results for combinations of constant or decreasing step sizes and constant or increasing sample/batch sizes. When all hyperparameters are fixed, NGVI is shown to converge geometrically to a neighborhood of the optimum, while we establish convergence to the optimum with rates of the form $\mathcal{O}\left(\frac{1}{T^{\rho}} \right)$, possibly with $\rho \geq 1$, for all other combinations of step size and sample/batch size schedules. These rates apply when the target posterior distribution is close in some sense to the considered exponential family. Our theoretical results extend existing NGVI and stochastic optimization results and provide more flexibility to adjust, in a principled way, step sizes and sample/batch sizes in order to meet speed, resources, or accuracy constraints.