The computational asymptotics of Gaussian variational inference and the Laplace approximation

TL;DR

This paper analyzes the asymptotic convexity of Gaussian variational inference and the Laplace approximation, proposing algorithms that reliably find global optima in large-sample regimes, improving Bayesian posterior approximation.

Contribution

It provides a theoretical analysis of asymptotic convexity properties and introduces CLA and CSVI algorithms that effectively find global optima in Gaussian variational inference and Laplace approximation.

Findings

CLA and CSVI outperform standard methods in likelihood of global optima

Algorithms reliably find the global optimum in asymptotic regimes

Experiments on synthetic and real data validate the approach

Abstract

Gaussian variational inference and the Laplace approximation are popular alternatives to Markov chain Monte Carlo that formulate Bayesian posterior inference as an optimization problem, enabling the use of simple and scalable stochastic optimization algorithms. However, a key limitation of both methods is that the solution to the optimization problem is typically not tractable to compute; even in simple settings the problem is nonconvex. Thus, recently developed statistical guarantees -- which all involve the (data) asymptotic properties of the global optimum -- are not reliably obtained in practice. In this work, we provide two major contributions: a theoretical analysis of the asymptotic convexity properties of variational inference with a Gaussian family and the maximum a posteriori (MAP) problem required by the Laplace approximation; and two algorithms -- consistent Laplace approximation (CLA) and consistent stochastic variational inference (CSVI) -- that exploit these properties to find the optimal approximation in the asymptotic regime. Both CLA and CSVI involve a tractable initialization procedure that finds the local basin of the optimum, and CSVI further includes a scaled gradient descent algorithm that provably stays locally confined to that basin. Experiments on nonconvex synthetic and real-data examples show that compared with standard variational and Laplace approximations, both CSVI and CLA improve the likelihood of obtaining the global optimum of their respective optimization problems.