Residual Belief Propagation for Topic Modeling

TL;DR

This paper introduces Residual Belief Propagation (RBP), a novel algorithm that accelerates LDA training by prioritizing fast-converging messages, resulting in faster convergence and lower perplexity compared to existing methods.

Contribution

The paper proposes RBP with an informed scheduling scheme for asynchronous message passing, significantly improving convergence speed and predictive performance in LDA training.

Findings

RBP reduces training time until convergence.

RBP achieves lower predictive perplexity.

RBP outperforms VB, GS, BP, and RVB in experiments.

Abstract

Fast convergence speed is a desired property for training latent Dirichlet allocation (LDA), especially in online and parallel topic modeling for massive data sets. This paper presents a novel residual belief propagation (RBP) algorithm to accelerate the convergence speed for training LDA. The proposed RBP uses an informed scheduling scheme for asynchronous message passing, which passes fast-convergent messages with a higher priority to influence those slow-convergent messages at each learning iteration. Extensive empirical studies confirm that RBP significantly reduces the training time until convergence while achieves a much lower predictive perplexity than other state-of-the-art training algorithms for LDA, including variational Bayes (VB), collapsed Gibbs sampling (GS), loopy belief propagation (BP), and residual VB (RVB).