Gaussian variational approximation with sparse precision matrices

TL;DR

This paper introduces a sparse precision matrix approach for Gaussian variational approximation, enabling efficient and flexible high-dimensional Bayesian inference with improved stability and applicability to complex models.

Contribution

It proposes a novel sparse precision matrix parameterization via Cholesky factors for Gaussian variational inference, with efficient stochastic gradient optimization methods.

Findings

Effective in generalized linear mixed models

Applicable to state space models for time series

Improves stability and efficiency of variational inference

Abstract

We consider the problem of learning a Gaussian variational approximation to the posterior distribution for a high-dimensional parameter, where we impose sparsity in the precision matrix to reflect appropriate conditional independence structure in the model. Incorporating sparsity in the precision matrix allows the Gaussian variational distribution to be both flexible and parsimonious, and the sparsity is achieved through parameterization in terms of the Cholesky factor. Efficient stochastic gradient methods which make appropriate use of gradient information for the target distribution are developed for the optimization. We consider alternative estimators of the stochastic gradients which have lower variation and are more stable. Our approach is illustrated using generalized linear mixed models and state space models for time series.