Covariance Matrices and Influence Scores for Mean Field Variational Bayes

TL;DR

This paper introduces Linear Response Variational Bayes (LRVB), a fast method to improve uncertainty and covariance estimates in mean field variational Bayes, applicable to large-scale exponential family models.

Contribution

It develops LRVB, a novel approach that augments MFVB to accurately estimate covariances and influence scores, addressing key limitations of existing variational methods.

Findings

LRVB provides more accurate uncertainty estimates than standard MFVB.

The method scales efficiently to large datasets and complex models.

LRVB accurately estimates influence of data points on parameters.

Abstract

Mean field variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, it is well known that a major failing of MFVB is that it underestimates the uncertainty of model variables (sometimes severely) and provides no information about model variable covariance. We develop a fast, general methodology for exponential families that augments MFVB to deliver accurate uncertainty estimates for model variables -- both for individual variables and coherently across variables. MFVB for exponential families defines a fixed-point equation in the means of the approximating posterior, and our approach yields a covariance estimate by perturbing this fixed point. Inspired by linear response theory, we call our method linear response variational Bayes (LRVB). We also show how LRVB can be used to quickly calculate a measure of the influence of individual data points on parameter point estimates. We demonstrate the accuracy and scalability of our method by learning Gaussian mixture models for both simulated and real data.