Hierarchical mixtures of Gaussians for combined dimensionality reduction and clustering

TL;DR

This paper presents hierarchical mixtures of Gaussians (HMoGs), a probabilistic model that combines dimensionality reduction and clustering, offering efficient inference, high-dimensional modeling, and improved performance on complex datasets.

Contribution

Introduction of HMoGs with exponential family parameterization, enabling efficient high-dimensional data modeling and joint optimization of reduction and clustering.

Findings

HMoGs outperform traditional methods on synthetic and MNIST data.

The model efficiently handles hundreds of latent dimensions.

Sparsity constraints enhance clustering and interpretability.

Abstract

We introduce hierarchical mixtures of Gaussians (HMoGs), which unify dimensionality reduction and clustering into a single probabilistic model. HMoGs provide closed-form expressions for the model likelihood, exact inference over latent states and cluster membership, and exact algorithms for maximum-likelihood optimization. The novel exponential family parameterization of HMoGs greatly reduces their computational complexity relative to similar model-based methods, allowing them to efficiently model hundreds of latent dimensions, and thereby capture additional structure in high-dimensional data. We demonstrate HMoGs on synthetic experiments and MNIST, and show how joint optimization of dimensionality reduction and clustering facilitates increased model performance. We also explore how sparsity-constrained dimensionality reduction can further improve clustering performance while encouraging interpretability. By bridging classical statistical modelling with the scale of modern data and compute, HMoGs offer a practical approach to high-dimensional clustering that preserves statistical rigour, interpretability, and uncertainty quantification that is often missing from embedding-based, variational, and self-supervised methods.