Blocked Gibbs sampler for hierarchical Dirichlet processes

TL;DR

This paper introduces a stable, scalable blocked Gibbs sampler for hierarchical Dirichlet process models, improving upon existing methods by using a truncated approximation and a gamma prior for better mixing and efficiency.

Contribution

It develops a novel blocked Gibbs sampler employing a truncated approximation and gamma prior, enhancing stability and scalability in HDP posterior inference.

Findings

The proposed sampler produces statistically stable results.

It is highly scalable with respect to sample size.

The sampler exhibits good mixing properties.

Abstract

Posterior computation in hierarchical Dirichlet process (HDP) mixture models is an active area of research in nonparametric Bayes inference of grouped data. Existing literature almost exclusively focuses on the Chinese restaurant franchise (CRF) analogy of the marginal distribution of the parameters, which can mix poorly and has a quadratic complexity with the sample size. A recently developed slice sampler allows for efficient blocked updates of the parameters, but is shown to be statistically unstable in our article. We develop a blocked Gibbs sampler that employs a truncated approximation of the underlying random measures to sample from the posterior distribution of HDP, which produces statistically stable results, is highly scalable with respect to sample size, and is shown to have good mixing. The heart of the construction is to endow the shared concentration parameter with an appropriately chosen gamma prior that allows us to break the dependence of the shared mixing proportions and permits independent updates of certain log-concave random variables in a block. En route, we develop an efficient rejection sampler for these random variables leveraging piece-wise tangent-line approximations.