How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Framework

TL;DR

This paper introduces a stochastic integral framework for analyzing discrete diffusion models, providing new theoretical insights, error bounds, and guidance for designing more efficient algorithms.

Contribution

It generalizes stochastic integral formulations for discrete diffusion models, establishing error bounds and unifying existing theoretical results.

Findings

First error bound for the -leaping scheme in KL divergence.

Unified theoretical framework for discrete diffusion models.

Insights into the mathematical properties and algorithm design for discrete diffusion.

Abstract

Discrete diffusion models have gained increasing attention for their ability to model complex distributions with tractable sampling and inference. However, the error analysis for discrete diffusion models remains less well-understood. In this work, we propose a comprehensive framework for the error analysis of discrete diffusion models based on L\'evy-type stochastic integrals. By generalizing the Poisson random measure to that with a time-independent and state-dependent intensity, we rigorously establish a stochastic integral formulation of discrete diffusion models and provide the corresponding change of measure theorems that are intriguingly analogous to It\^o integrals and Girsanov's theorem for their continuous counterparts. Our framework unifies and strengthens the current theoretical results on discrete diffusion models and obtains the first error bound for the $\tau$-leaping scheme in KL divergence. With error sources clearly identified, our analysis gives new insight into the mathematical properties of discrete diffusion models and offers guidance for the design of efficient and accurate algorithms for real-world discrete diffusion model applications.