Moment Propagation

TL;DR

Moment propagation is a new approximate Bayesian inference method that corrects variance underestimation in mean field variational Bayes, with proven accuracy for certain models and empirical success on benchmark datasets.

Contribution

The paper introduces moment propagation, a novel variance correction technique for mean field variational Bayes, applicable to linear, normal, and probit regression models.

Findings

Recovers exact marginal posteriors for linear and normal models.

Provides asymptotically correct means and covariances for probit regression.

Performs well empirically on benchmark datasets.

Abstract

We introduce and develop moment propagation for approximate Bayesian inference. This method can be viewed as a variance correction for mean field variational Bayes which tends to underestimate posterior variances. Focusing on the case where the model is described by two sets of parameter vectors, we develop moment propagation algorithms for linear regression, multivariate normal, and probit regression models. We show for the probit regression model that moment propagation empirically performs reasonably well for several benchmark datasets. Finally, we discuss theoretical gaps and future extensions. In the supplementary material we show heuristically why moment propagation leads to appropriate posterior variance estimation, for the linear regression and multivariate normal models we show precisely why mean field variational Bayes underestimates certain moments, and prove that our moment propagation algorithm recovers the exact marginal posterior distributions for all parameters, and for probit regression we show that moment propagation provides asymptotically correct posterior means and covariance estimates.