Variational Inference for Latent Variable Models in High Dimensions

TL;DR

This paper develops a framework to evaluate the accuracy of mean-field variational inference in high-dimensional Bayesian latent variable models, demonstrating its effectiveness and limitations through specific models like LDA and stochastic blockmodels.

Contribution

It introduces a general framework for analyzing the statistical accuracy of MFVI in high-dimensional models, including new algorithms and finite-sample performance bounds.

Findings

MFVI's exact regime of effectiveness in LDA identified.

Partially grouped VI improves performance in stochastic blockmodels.

Bounds are shown to be tight for the studied models.

Abstract

Variational inference (VI) is a popular method for approximating intractable posterior distributions in Bayesian inference and probabilistic machine learning. In this paper, we introduce a general framework for quantifying the statistical accuracy of mean-field variational inference (MFVI) for posterior approximation in Bayesian latent variable models with categorical local latent variables (and arbitrary global latent variables). Utilizing our general framework, we capture the exact regime where MFVI 'works' for the celebrated latent Dirichlet allocation model. Focusing on the mixed membership stochastic blockmodel, we show that the vanilla fully factorized MFVI, often used in the literature, is suboptimal. We propose a partially grouped VI algorithm for this model and show that it works, and derive its exact finite-sample performance. We further illustrate that our bounds are tight for both the above models. Our proof techniques, which extend the framework of nonlinear large deviations, open the door for the analysis of MFVI in other latent variable models.