Approximate Collapsed Gibbs Clustering with Expectation Propagation

TL;DR

This paper introduces an expectation propagation-based approximation method for collapsed Gibbs sampling in complex latent variable models, enabling faster inference with maintained accuracy.

Contribution

It presents a novel framework that approximates collapsed Gibbs sampling using expectation propagation, applicable to models where exact integration is infeasible.

Findings

Enables efficient sampling in intractable models like Student-t mixtures and time series clustering.

Provides a runtime-accuracy tradeoff for approximate collapsed Gibbs sampling.

Achieves competitive accuracy with significantly reduced computation time.

Abstract

We develop a framework for approximating collapsed Gibbs sampling in generative latent variable cluster models. Collapsed Gibbs is a popular MCMC method, which integrates out variables in the posterior to improve mixing. Unfortunately for many complex models, integrating out these variables is either analytically or computationally intractable. We efficiently approximate the necessary collapsed Gibbs integrals by borrowing ideas from expectation propagation. We present two case studies where exact collapsed Gibbs sampling is intractable: mixtures of Student-t's and time series clustering. Our experiments on real and synthetic data show that our approximate sampler enables a runtime-accuracy tradeoff in sampling these types of models, providing results with competitive accuracy much more rapidly than the naive Gibbs samplers one would otherwise rely on in these scenarios.