Blocked Gibbs Sampling for Improved Convergence in Finite Mixture Models

TL;DR

This paper introduces a blocked Gibbs sampling method for finite mixture models that reduces autocorrelation and improves convergence speed by addressing correlations among latent variables, especially for outliers.

Contribution

It analyzes latent variable correlations in mixture models and proposes a blocking strategy that enhances Gibbs sampling efficiency.

Findings

Blocked sampling reduces autocorrelation in latent variables.

The method improves convergence speed in simulations.

Outlier latent variables benefit from the proposed blocking approach.

Abstract

Gibbs sampling is a common procedure used to fit finite mixture models. However, it is known to be slow to converge when exploring correlated regions of a parameter space and so blocking correlated parameters is sometimes implemented in practice. This is straightforward to visualize in contexts like low-dimensional multivariate Gaussian distributions, but more difficult for mixture models because of the way latent variable assignment and cluster-specific parameters influence one another. Here we analyze correlation in the space of latent variables and show that latent variables of outlier observations equidistant between component distributions can exhibit significant correlation that is not bounded away from one, suggesting they can converge very slowly to their stationary distribution. We provide bounds on convergence rates to a modification of the stationary distribution and propose a blocked sampling procedure that significantly reduces autocorrelation in the latent variable Markov chain, which we demonstrate in simulation.