Gibbs sampling for mixtures in order of appearance: the ordered allocation sampler

TL;DR

This paper introduces an ordered allocation Gibbs sampler for mixture models that improves mixing and avoids truncation, especially in infinite and variable-dimension mixtures, by leveraging the order of appearance of components.

Contribution

The paper presents a novel Gibbs sampling method based on ordered components, enhancing efficiency and applicability in complex mixture models without truncation.

Findings

Effective in infinite mixtures without truncation

Improves mixing properties over traditional samplers

Eases trans-dimensional sampling in finite mixtures

Abstract

Gibbs sampling methods are standard tools to perform posterior inference for mixture models. These have been broadly classified into two categories: marginal and conditional methods. While conditional samplers are more widely applicable than marginal ones, they may suffer from slow mixing in infinite mixtures, where some form of truncation, either deterministic or random, is required. In mixtures with random number of components, the exploration of parameter spaces of different dimensions can also be challenging. We tackle these issues by expressing the mixture components in the random order of appearance in an exchangeable sequence directed by the mixing distribution. We derive a sampler that is straightforward to implement for mixing distributions with tractable size-biased ordered weights, and that can be readily adapted to mixture models for which marginal samplers are not available. In infinite mixtures, no form of truncation is necessary. As for finite mixtures with random dimension, a simple updating of the number of components is obtained by a blocking argument, thus, easing challenges found in trans-dimensional moves via Metropolis-Hastings steps. Additionally, sampling occurs in the space of ordered partitions with blocks labelled in the least element order, which endows the sampler with good mixing properties. The performance of the proposed algorithm is evaluated in a simulation study.