Incremental Variational Inference for Latent Dirichlet Allocation

TL;DR

This paper presents incremental variational inference for LDA, enabling faster convergence and scalability to large datasets, with a novel distributed version that maintains performance while significantly increasing speed.

Contribution

It introduces incremental variational inference for LDA, offering an alternative to stochastic methods with faster convergence and a new distributed algorithm for large-scale topic modeling.

Findings

Incremental LDA converges faster and monotonically increases the variational bound.

The distributed algorithm achieves comparable performance with significant speed-up.

The method effectively processes massive document collections without learning rate tuning.

Abstract

We introduce incremental variational inference and apply it to latent Dirichlet allocation (LDA). Incremental variational inference is inspired by incremental EM and provides an alternative to stochastic variational inference. Incremental LDA can process massive document collections, does not require to set a learning rate, converges faster to a local optimum of the variational bound and enjoys the attractive property of monotonically increasing it. We study the performance of incremental LDA on large benchmark data sets. We further introduce a stochastic approximation of incremental variational inference which extends to the asynchronous distributed setting. The resulting distributed algorithm achieves comparable performance as single host incremental variational inference, but with a significant speed-up.