Tightening Bounds for Variational Inference by Revisiting Perturbation Theory

TL;DR

This paper improves variational inference by developing a new perturbation-based bound that is tighter and more mass-covering, leading to better posterior approximations and higher data likelihoods.

Contribution

It introduces a novel correction method inspired by perturbation theory that yields valid bounds for variational inference, enhancing approximation quality.

Findings

New bounds are more mass covering.

Posterior covariances are closer to true posteriors.

Higher likelihoods on held-out data.

Abstract

Variational inference has become one of the most widely used methods in latent variable modeling. In its basic form, variational inference employs a fully factorized variational distribution and minimizes its KL divergence to the posterior. As the minimization can only be carried out approximately, this approximation induces a bias. In this paper, we revisit perturbation theory as a powerful way of improving the variational approximation. Perturbation theory relies on a form of Taylor expansion of the log marginal likelihood, vaguely in terms of the log ratio of the true posterior and its variational approximation. While first order terms give the classical variational bound, higher-order terms yield corrections that tighten it. However, traditional perturbation theory does not provide a lower bound, making it inapt for stochastic optimization. In this paper, we present a similar yet alternative way of deriving corrections to the ELBO that resemble perturbation theory, but that result in a valid bound. We show in experiments on Gaussian Processes and Variational Autoencoders that the new bounds are more mass covering, and that the resulting posterior covariances are closer to the true posterior and lead to higher likelihoods on held-out data.