The Discrete Infinite Logistic Normal Distribution

TL;DR

The paper introduces the discrete infinite logistic normal distribution (DILN), a Bayesian nonparametric prior that models correlations in mixed membership models, with applications to topic modeling and scalable inference.

Contribution

It generalizes the hierarchical Dirichlet process by modeling correlation structures and develops variational and online inference algorithms for large-scale data.

Findings

DILN outperforms HDP and CTM in topic modeling tasks.

The online inference algorithm scales efficiently to large datasets.

Empirical results demonstrate improved modeling of correlations.

Abstract

We present the discrete infinite logistic normal distribution (DILN), a Bayesian nonparametric prior for mixed membership models. DILN is a generalization of the hierarchical Dirichlet process (HDP) that models correlation structure between the weights of the atoms at the group level. We derive a representation of DILN as a normalized collection of gamma-distributed random variables, and study its statistical properties. We consider applications to topic modeling and derive a variational inference algorithm for approximate posterior inference. We study the empirical performance of the DILN topic model on four corpora, comparing performance with the HDP and the correlated topic model (CTM). To deal with large-scale data sets, we also develop an online inference algorithm for DILN and compare with online HDP and online LDA on the Nature magazine, which contains approximately 350,000 articles.