Measuring Temporal Linguistic Emergence in Diffusion Language Models

TL;DR

This paper investigates the temporal dynamics of information emergence in diffusion language models, revealing patterns of stability, recoverability, and sensitivity during the denoising process.

Contribution

It introduces a set of temporal measurements to analyze information emergence in diffusion language models, providing new insights into their generative trajectories.

Findings

Content categories stabilize earlier than function words.

POS and semantic labels are more linearly recoverable than exact tokens.

Uncertainty correlates with eventual correctness and peaks mid-trajectory.

Abstract

Diffusion language models expose an explicit denoising trajectory, making it possible to ask when different kinds of information become measurable during generation. We study three independent 32-step runs of LLaDA-8B-Base on masked WikiText-103 text, each with 1{,}000 probe-training sequences and 200 held-out evaluation sequences. From saved trajectories, we derive four temporal measurements: token commitment; linear recoverability of part-of-speech (POS), coarse semantic category, and token identity; confidence and entropy dynamics; and sensitivity under mid-trajectory re-masking. Across seeds, the same ordering recurs: content categories stabilize earlier than function-heavy categories, POS and coarse semantic labels remain substantially more linearly recoverable than exact lexical identity under our probe setup, uncertainty remains higher for tokens that ultimately resolve incorrectly even though late confidence becomes less calibrated, and perturbation sensitivity peaks in the middle of the trajectory. A direct/collateral decomposition shows that this peak is overwhelmingly local to the perturbed positions themselves. In this LLaDA+WikiText setting, denoising time is therefore a useful analysis axis: under our measurements, coarse labels are recovered earlier and more robustly than lexical identity, trajectory-level uncertainty tracks eventual correctness, and mid-trajectory states are the most intervention-sensitive.