Covariances, Robustness, and Variational Bayes

TL;DR

This paper enhances mean-field Variational Bayes by deriving formulas for improved covariance estimates and robustness measures, enabling faster and more reliable uncertainty quantification in large-scale Bayesian inference.

Contribution

It introduces a simple formula linking model perturbations to covariance estimates and robustness measures, expanding MFVB's practical utility.

Findings

Provides accurate covariance estimates for MFVB

Offers local robustness measures for model perturbations

Achieves faster runtimes compared to MCMC

Abstract

Mean-field Variational Bayes (MFVB) is an approximate Bayesian posterior inference technique that is increasingly popular due to its fast runtimes on large-scale datasets. However, even when MFVB provides accurate posterior means for certain parameters, it often mis-estimates variances and covariances. Furthermore, prior robustness measures have remained undeveloped for MFVB. By deriving a simple formula for the effect of infinitesimal model perturbations on MFVB posterior means, we provide both improved covariance estimates and local robustness measures for MFVB, thus greatly expanding the practical usefulness of MFVB posterior approximations. The estimates for MFVB posterior covariances rely on a result from the classical Bayesian robustness literature relating derivatives of posterior expectations to posterior covariances and include the Laplace approximation as a special case. Our key condition is that the MFVB approximation provides good estimates of a select subset of posterior means---an assumption that has been shown to hold in many practical settings. In our experiments, we demonstrate that our methods are simple, general, and fast, providing accurate posterior uncertainty estimates and robustness measures with runtimes that can be an order of magnitude faster than MCMC.