A derivation of variational message passing (VMP) for latent Dirichlet allocation (LDA)

TL;DR

This paper provides a detailed derivation of variational message passing (VMP) update equations specifically for Latent Dirichlet Allocation (LDA), facilitating easier implementation of VMP for topic models.

Contribution

It offers the first comprehensive derivation of VMP updates for LDA, addressing a gap in practical implementation guidance.

Findings

Provides step-by-step derivation of VMP for LDA

Enables easier implementation of VMP in probabilistic graphical models

Facilitates future research on variational inference methods

Abstract

Latent Dirichlet Allocation (LDA) is a probabilistic model used to uncover latent topics in a corpus of documents. Inference is often performed using variational Bayes (VB) algorithms, which calculate a lower bound to the posterior distribution over the parameters. Deriving the variational update equations for new models requires considerable manual effort; variational message passing (VMP) has emerged as a "black-box" tool to expedite the process of variational inference. But applying VMP in practice still presents subtle challenges, and the existing literature does not contain the steps that are necessary to implement VMP for the standard smoothed LDA model, nor are available black-box probabilistic graphical modelling software able to do the word-topic updates necessary to implement LDA. In this paper, we therefore present a detailed derivation of the VMP update equations for LDA. We see this as a first step to enabling other researchers to calculate the VMP updates for similar graphical models.