Fast Variational Inference in the Conjugate Exponential Family

TL;DR

This paper introduces a unified, faster variational inference method for conjugate exponential family models, leveraging information geometry to improve optimization speed and efficiency.

Contribution

It presents a general framework for collapsed variational inference that unifies existing methods and introduces a new lower bound with accelerated optimization techniques.

Findings

Significant speed-ups in probabilistic model optimization

A new lower bound on marginal likelihood

Applicability to a wide range of models

Abstract

We present a general method for deriving collapsed variational inference algo- rithms for probabilistic models in the conjugate exponential family. Our method unifies many existing approaches to collapsed variational inference. Our collapsed variational inference leads to a new lower bound on the marginal likelihood. We exploit the information geometry of the bound to derive much faster optimization methods based on conjugate gradients for these models. Our approach is very general and is easily applied to any model where the mean field update equations have been derived. Empirically we show significant speed-ups for probabilistic models optimized using our bound.