Exact slice sampler for Hierarchical Dirichlet Processes

TL;DR

This paper introduces an exact slice sampling algorithm for Hierarchical Dirichlet Processes (HDPs), enabling efficient, parallelizable inference while automatically truncating infinite measures without approximation.

Contribution

The authors develop a novel Bayesian variable augmentation approach to create a full slice sampler for HDPs, addressing previous limitations and enabling exact, efficient sampling.

Findings

Fast mixing of the sampler.

Exact inference with natural truncation.

Suitable for parallel implementation.

Abstract

We propose an exact slice sampler for Hierarchical Dirichlet process (HDP) and its associated mixture models (Teh et al., 2006). Although there are existing MCMC algorithms for sampling from the HDP, a slice sampler has been missing from the literature. Slice sampling is well-known for its desirable properties including its fast mixing and its natural potential for parallelization. On the other hand, the hierarchical nature of HDPs poses challenges to adopting a full-fledged slice sampler that automatically truncates all the infinite measures involved without ad-hoc modifications. In this work, we adopt the powerful idea of Bayesian variable augmentation to address this challenge. By introducing new latent variables, we obtain a full factorization of the joint distribution that is suitable for slice sampling. Our algorithm has several appealing features such as (1) fast mixing; (2) remaining exact while allowing natural truncation of the underlying infinite-dimensional measures, as in (Kalli et al., 2011), resulting in updates of only a finite number of necessary atoms and weights in each iteration; and (3) being naturally suited to parallel implementations. The underlying principle for joint factorization of the full likelihood is simple and can be applied to many other settings, such as designing sampling algorithms for general dependent Dirichlet process (DDP) models.