Sharp Convergence Rates for Masked Diffusion Models

TL;DR

This paper provides new theoretical convergence guarantees for masked diffusion models, especially the Euler and FHS samplers, using total variation analysis to improve understanding and establish tight bounds.

Contribution

It introduces a TV-based analysis for the Euler method that relaxes assumptions and provides tight bounds, and offers the first lower bound for Euler convergence and an analysis of FHS error.

Findings

Euler method convergence bounds are improved and assumptions relaxed.

FHS incurs no additional sampling error beyond score estimation.

Established tight lower bounds for both Euler and FHS samplers.

Abstract

Discrete diffusion models have achieved strong empirical performance in text and other symbolic domains, with masked (absorbing-rate) variants emerging as competitive alternatives to autoregressive models. Among existing samplers, the Euler method remains the standard choice in many applications, and more recently, the First-Hitting Sampler (FHS) has shown considerable promise for masked diffusion models. Despite their practical success, the theoretical understanding of these samplers remains limited. Existing analyses are conducted in Kullback-Leibler (KL) divergence, which often yields loose parameter dependencies and requires strong assumptions on score estimation. Moreover, these guarantees do not cover recently developed high-performance sampler of FHS. In this work, we first develop a direct total-variation (TV) based analysis for the Euler method that overcomes these limitations. Our results relax assumptions on score estimation, improve parameter dependencies, and establish convergence guarantees without requiring any surrogate initialization. Also for this setting, we provide the first convergence lower bound for the Euler sampler, establishing tightness with respect to both the data dimension $d$ and the target accuracy $\varepsilon$. Finally, we analyze the FHS sampler and show that it incurs no sampling error beyond that induced by score estimation, which we show to be tight with a matching lower error bound. Overall, our analysis introduces a direct TV-based error decomposition along the CTMC trajectory and a decoupling-based path-wise analysis for FHS, which may be of independent interest.