Path-Dependent Denoising: A Non-Conservative Field Perspective on Order Collapse in Diffusion Language Models

TL;DR

This paper introduces a theoretical framework for understanding order sensitivity in diffusion language models, focusing on local denoising conditionals and their compatibility to enable order-invariant decoding.

Contribution

It formalizes the concept of order-induced pseudo-joints and circulation, providing diagnostics to test when DLM decoding is truly order-free.

Findings

Order sensitivity in DLM decoding stems from incompatibility of local denoising conditionals.

The framework decomposes path dependence into local circulations and errors.

Diagnostics can identify when DLM decoding is genuinely order-invariant.

Abstract

Diffusion language models (DLMs) offer a structural alternative to autoregressive generation: denoising can update tokens in arbitrary orders or in parallel rather than along a fixed left-to-right chain. In practice, fast DLM decoding remains strongly order-sensitive and often drifts toward autoregressive-like trajectories. We trace this tension to compatibility. At each reverse-time step, a DLM provides local denoising conditionals over the unresolved tokens. Arbitrary-order denoising becomes well defined when these local conditionals compose into order-invariant pseudo-joints. We formalize this view by defining order-induced pseudo-joints and a local denoising circulation: the log-ratio between the two pseudo-joints obtained by swapping a pair of unresolved positions. This circulation is zero under compatible conditionals, and global order gaps decompose into sums of local circulations along adjacent swaps. We further separate incompatibility-driven path dependence from conditional-dependence error in parallel updates and from order-specific estimation error. The resulting framework provides inference-only diagnostics for testing when DLM decoding is genuinely order-free.