Exact inference via quasi-conjugacy in two-parameter Poisson-Dirichlet hidden Markov models

TL;DR

This paper presents a novel nonparametric inference framework for time-evolving unobserved distributions using a two-parameter Poisson-Dirichlet diffusion model, providing exact posterior and predictive distributions without MCMC.

Contribution

It introduces a tractable recursive inference method leveraging duality and coagulation operators, enabling exact, efficient inference in complex partition models.

Findings

Exact posterior distributions are computed efficiently.

The method outperforms particle filtering in accuracy and computational efficiency.

Applications demonstrate interpretable insights into social and genetic data.

Abstract

We introduce a nonparametric model for inferring time-evolving, unobserved probability distributions from discrete-time data consisting of unlabelled partitions. The latent process is a two-parameter Poisson-Dirichlet diffusion, and observations arise via exchangeable sampling. Applications include social and genetic data where only aggregate clustering summaries are observed. To address the intractable likelihood, we develop a tractable inferential framework that avoids label enumeration and direct simulation of the latent state. We exploit a duality between the diffusion and a pure-death process on partitions, together with coagulation operators that encode the effect of new data. These yield closed-form, recursive updates for forward and backward inference. We compute exact posterior distributions of the latent state at arbitrary times and predictive distributions of future or interpolated partitions. This enables online and offline inference and forecasting with full uncertainty quantification, bypassing MCMC and sequential Monte Carlo. Compared to particle filtering, our method achieves higher accuracy, lower variance, and substantial computational gains. We illustrate the methodology with synthetic experiments and a social network application, recovering interpretable patterns in time-varying heterozygosity.