Hierarchical Dirichlet Process and Relative Entropy

TL;DR

This paper analyzes the theoretical properties of the hierarchical Dirichlet process, deriving laws of large numbers and large deviations, and explicitly identifying rate functions related to relative entropy at multiple levels.

Contribution

It provides the first rigorous analysis of the asymptotic behavior of the hierarchical Dirichlet process, including explicit large deviation rate functions.

Findings

Large deviation rate functions are explicitly derived.

Rate function includes two relative entropy terms at different levels.

Hierarchical Dirichlet process has slower cluster growth than standard Dirichlet process.

Abstract

The Hierarchical Dirichlet process is a discrete random measure serving as an important prior in Bayesian non-parametrics. It is motivated with the study of groups of clustered data. Each group is modelled through a level two Dirichlet process and all groups share the same base distribution which itself is a drawn from a level one Dirichlet process. It has two concentration parameters with one at each level. The main results of the paper are the law of large numbers and large deviations for the hierarchical Dirichlet process and its mass when both concentration parameters converge to infinity. The large deviation rate functions are identified explicitly. The rate function for the hierarchical Dirichlet process consists of two terms corresponding to the relative entropies at each level. It is less than the rate function for the Dirichlet process, which reflects the fact that the number of clusters under the hierarchical Dirichlet process has a slower growth rate than under the Dirichlet process.