What Exactly Does Guidance Do in Masked Discrete Diffusion Models

TL;DR

This paper provides a precise theoretical analysis of how classifier-free guidance influences the sampling process in masked discrete diffusion models, revealing its effects on distribution shaping and sampling dynamics.

Contribution

We derive an explicit solution for guided reverse dynamics in masked discrete diffusion models, characterizing guidance's influence on sampling behavior and distribution structure.

Findings

Guidance amplifies class-specific regions in the distribution.

Total variation decay rate is double-exponential in guidance strength.

Guidance affects both output distribution and sampling trajectory dynamics.

Abstract

We study masked discrete diffusion models with classifier-free guidance (CFG). Assuming no score error nor discretization error, we derive an explicit solution to the guided reverse dynamics, so that how guidance influences the sampling behavior can be precisely characterized. When the full data distribution is a mixture over classes and the goal is to sample from a specific class, guidance amplifies class-specific regions while suppresses regions shared with other classes. This effect depends on the guidance strength $w$ and induces distinct covariance structures in the sampled distribution. Notably, we observe quantitatively different behaviors in $1$D and $2$D. We also show that for large $w$, the decay rate of the total variation ($\mathrm{TV}$) along the reverse dynamics is double-exponential in $w$ for both $1$D and $2$D. These findings highlight the role of guidance, not just in shaping the output distribution, but also in controlling the dynamics of the sampling trajectory. Our theoretical analysis is supported by experiments that illustrate the geometric effects of guidance and its impact on convergence.