Learning Topic Models by Belief Propagation

TL;DR

This paper introduces a belief propagation approach for learning latent Dirichlet allocation (LDA), demonstrating competitive speed and accuracy, and extends it to variants like author-topic and relational topic models.

Contribution

It presents a novel BP-based inference method for LDA and its variants, offering an alternative to traditional methods like VB and Gibbs sampling.

Findings

BP is competitive in speed and accuracy compared to VB and Gibbs sampling.

BP can be extended to learn variants like ATM and RTM.

Experimental results on large datasets validate the effectiveness of BP.

Abstract

Latent Dirichlet allocation (LDA) is an important hierarchical Bayesian model for probabilistic topic modeling, which attracts worldwide interests and touches on many important applications in text mining, computer vision and computational biology. This paper represents LDA as a factor graph within the Markov random field (MRF) framework, which enables the classic loopy belief propagation (BP) algorithm for approximate inference and parameter estimation. Although two commonly-used approximate inference methods, such as variational Bayes (VB) and collapsed Gibbs sampling (GS), have gained great successes in learning LDA, the proposed BP is competitive in both speed and accuracy as validated by encouraging experimental results on four large-scale document data sets. Furthermore, the BP algorithm has the potential to become a generic learning scheme for variants of LDA-based topic models. To this end, we show how to learn two typical variants of LDA-based topic models, such as author-topic models (ATM) and relational topic models (RTM), using BP based on the factor graph representation.