Streaming Variational Bayes

TL;DR

This paper introduces SDA-Bayes, a flexible framework for streaming, distributed, asynchronous Bayesian inference, demonstrated with variational Bayes on large-scale text data, outperforming stochastic variational inference in streaming scenarios.

Contribution

The paper proposes SDA-Bayes, a novel framework enabling streaming and distributed Bayesian updates, extending variational Bayes to streaming data and surpassing SVI in certain settings.

Findings

SDA-Bayes effectively fits LDA to large datasets.

It outperforms stochastic variational inference in streaming contexts.

The framework supports asynchronous, distributed computation.

Abstract

We present SDA-Bayes, a framework for (S)treaming, (D)istributed, (A)synchronous computation of a Bayesian posterior. The framework makes streaming updates to the estimated posterior according to a user-specified approximation batch primitive. We demonstrate the usefulness of our framework, with variational Bayes (VB) as the primitive, by fitting the latent Dirichlet allocation model to two large-scale document collections. We demonstrate the advantages of our algorithm over stochastic variational inference (SVI) by comparing the two after a single pass through a known amount of data---a case where SVI may be applied---and in the streaming setting, where SVI does not apply.