Variational Boosting: Iteratively Refining Posterior Approximations

TL;DR

Variational boosting is a flexible, iterative black-box variational inference method that refines posterior approximations by expanding the variational class, improving accuracy and efficiency in statistical models.

Contribution

It introduces a novel iterative approach to variational inference that adaptively enhances the approximation class, balancing computational cost and accuracy.

Findings

Outperforms existing methods in accuracy

Provides more flexible posterior approximations

Efficient in synthetic and real models

Abstract

We propose a black-box variational inference method to approximate intractable distributions with an increasingly rich approximating class. Our method, termed variational boosting, iteratively refines an existing variational approximation by solving a sequence of optimization problems, allowing the practitioner to trade computation time for accuracy. We show how to expand the variational approximating class by incorporating additional covariance structure and by introducing new components to form a mixture. We apply variational boosting to synthetic and real statistical models, and show that resulting posterior inferences compare favorably to existing posterior approximation algorithms in both accuracy and efficiency.