Covariance Matrices for Mean Field Variational Bayes

TL;DR

This paper introduces Linear Response Variational Bayes (LRVB), a fast method to improve uncertainty and covariance estimates in Mean Field Variational Bayes for exponential family models, addressing its known limitations.

Contribution

It develops a general, efficient approach to augment MFVB with accurate covariance estimates using linear response theory, enhancing uncertainty quantification.

Findings

LRVB provides more accurate covariance estimates than standard MFVB.

The method is fast and applicable to large-scale data sets.

Demonstrated effectiveness on simulated data.

Abstract

Mean Field Variational Bayes (MFVB) is a popular posterior approximation method due to its fast runtime on large-scale data sets. However, it is well known that a major failing of MFVB is its (sometimes severe) underestimates of the uncertainty of model variables and lack of information about model variable covariance. We develop a fast, general methodology for exponential families that augments MFVB to deliver accurate uncertainty estimates for model variables -- both for individual variables and coherently across variables. MFVB for exponential families defines a fixed-point equation in the means of the approximating posterior, and our approach yields a covariance estimate by perturbing this fixed point. Inspired by linear response theory, we call our method linear response variational Bayes (LRVB). We demonstrate the accuracy of our method on simulated data sets.