Dynamical Regimes of Discrete Diffusion Models

TL;DR

This paper analyzes the backward dynamics of discrete diffusion models for data generation, identifying phase transitions that influence sample quality and diversity, and confirms theoretical predictions with simulations and experiments.

Contribution

It extends the theoretical framework of diffusion model phase transitions from continuous to discrete data, using statistical mechanics methods.

Findings

Speciation transition is a second-order phase transition analyzed via high-temperature expansion.

Collapse transition corresponds to a condensation transition described by the Random Energy Model.

Analytical speciation time matches empirical scaling when noise increases with time.

Abstract

Diffusion models generate high-dimensional data such as images by learning a process that gradually removes noise from corrupted data. Recent studies have shown that the backward dynamics of diffusion models exhibit two characteristic transitions: the speciation transition, at which generated samples begin to capture the global structure of the training data, and the collapse transition, at which the generation dynamics starts committing to individual training samples. While these transitions have been theoretically analyzed for continuous data, the same theoretical criteria have not been applied for discrete diffusion models, which are diffusion models for discrete data. In this work, we propose a simple effective model for discrete diffusion models trained on two-class Ising variable data with a general mixture ratio and analyze its backward dynamics using methods from statistical mechanics. We show that, as in the previous study on continuous data, the speciation transition can be determined through a second-order phase transition analysis using high-temperature expansion, while the collapse transition corresponds to a condensation transition described by the Random Energy Model. An analytical expression for the speciation time is obtained, and we show that its scaling becomes consistent with that of the continuous case when the noise increases with time as in practical diffusion models. These theoretical predictions are confirmed by numerical simulations and experiments with trained discrete diffusion models on real datasets. These results suggest that the original theoretical framework for continuous data remain valid for discrete data, and may provide a useful starting point for the statistical-mechanics analysis of discrete generative diffusion in more realistic settings.