A Split-Merge MCMC Algorithm for the Hierarchical Dirichlet Process

TL;DR

This paper introduces a novel split-merge MCMC algorithm for the hierarchical Dirichlet process, improving posterior inference efficiency for grouped data models like topic models.

Contribution

It develops a new split-merge MCMC method tailored for the HDP, enhancing inference over traditional Gibbs sampling methods.

Findings

Split-merge MCMC outperforms Gibbs sampling in synthetic and real data.

The algorithm provides faster convergence and better mixing.

Data properties influence the extent of improvement.

Abstract

The hierarchical Dirichlet process (HDP) has become an important Bayesian nonparametric model for grouped data, such as document collections. The HDP is used to construct a flexible mixed-membership model where the number of components is determined by the data. As for most Bayesian nonparametric models, exact posterior inference is intractable---practitioners use Markov chain Monte Carlo (MCMC) or variational inference. Inspired by the split-merge MCMC algorithm for the Dirichlet process (DP) mixture model, we describe a novel split-merge MCMC sampling algorithm for posterior inference in the HDP. We study its properties on both synthetic data and text corpora. We find that split-merge MCMC for the HDP can provide significant improvements over traditional Gibbs sampling, and we give some understanding of the data properties that give rise to larger improvements.