Support Before Frequency in Discrete Diffusion

TL;DR

This paper shows that discrete diffusion models learn the structure of data support before mastering the finer details of data frequencies, with implications for their denoising objectives and learning hierarchy.

Contribution

It provides a theoretical analysis of how different diffusion mechanisms organize learning, highlighting the support-frequency hierarchy in discrete diffusion models.

Findings

Support structure is learned before frequency details.

Uniform and absorbing diffusion exhibit different validity-improving behaviors.

Experiments confirm the predicted hierarchy and mechanism-dependent differences.

Abstract

Discrete diffusion models are increasingly competitive for language modeling, yet it remains unclear how their denoising objectives organize learning. Although these objectives target the full data distribution, we show that the exact reverse process induces a hierarchy between coarse support information and finer frequency information. For uniform and absorbing (a.k.a. masking) diffusion, we prove that, in the small-noise regime of the final denoising steps, each single-token reverse edit decomposes into a leading scale, determined by whether it moves toward the data support (e.g., grammatically valid sentences), and a finer coefficient, determining relative probabilities within the same scale. Thus, recovering validity structure only requires learning the correct order of magnitude of reverse probabilities, whereas recovering data frequencies requires coefficient-level estimation. The separation is mechanism-dependent: uniform diffusion exhibits a trichotomy into validity-improving, validity-preserving, and validity-worsening edits, while absorbing diffusion places its leading-order mass on validity-improving moves. Experiments on a masked language diffusion model and synthetic regular-language tasks support these predictions: support-localization emerges earlier than within-support frequency ranking, and the contrast between uniform and absorbing diffusion matches the predicted rate separation. Together, our results suggest that discrete diffusion models learn data support before data frequencies.