Hierarchical mixtures of Unigram models for short text clustering: The role of Beta-Liouville priors

TL;DR

This paper introduces a novel hierarchical mixture model for short text clustering using Beta-Liouville priors, offering more flexible correlation modeling and scalable inference methods, with demonstrated empirical effectiveness.

Contribution

It proposes a Beta-Liouville prior for multinomial mixtures, deriving conjugacy properties and scalable variational inference algorithms for short text clustering.

Findings

Beta-Liouville prior provides flexible correlation modeling.

Derived conjugacy enables efficient variational inference.

Empirical results show improved clustering performance.

Abstract

This paper presents a variant of the Multinomial mixture model tailored to the unsupervised classification of short text data. While the Multinomial probability vector is traditionally assigned a Dirichlet prior distribution, this work explores an alternative formulation based on the Beta-Liouville distribution, which offers a more flexible correlation structure than the Dirichlet. We examine the theoretical properties of the Beta-Liouville distribution, with particular focus on its conjugacy with the Multinomial likelihood. This property enables the derivation of update equations for a CAVI (Coordinate Ascent Variational Inference) algorithm, facilitating approximate posterior inference of the model parameters. In addition, we introduce a stochastic variant of the CAVI algorithm to enhance scalability. The paper concludes with empirical examples demonstrating effective strategies for selecting the Beta-Liouville hyperparameters.