Breaking the Factorization Barrier in Diffusion Language Models

TL;DR

This paper introduces CoDD, a hybrid diffusion framework that overcomes the factorization barrier in language models by modeling complex joint dependencies efficiently, improving coherence and speed.

Contribution

The paper proposes Coupled Discrete Diffusion (CoDD), a novel method that replaces fully-factorized outputs with a lightweight, expressive probabilistic layer, enabling joint dependency modeling without parameter explosion.

Findings

CoDD enhances diverse diffusion models with negligible overhead.

It matches the reasoning performance of RL baselines at lower training costs.

Prevents performance collapse in few-step generation, reducing latency.

Abstract

Diffusion language models theoretically allow for efficient parallel generation but are practically hindered by the "factorization barrier": the assumption that simultaneously predicted tokens are independent. This limitation forces a trade-off: models must either sacrifice speed by resolving dependencies sequentially or suffer from incoherence due to factorization. We argue that this barrier arises not from limited backbone expressivity, but from a structural misspecification: models are restricted to fully factorized outputs because explicitly parameterizing a joint distribution would require the Transformer to output a prohibitively large number of parameters. We propose Coupled Discrete Diffusion (CoDD), a hybrid framework that breaks this barrier by replacing the fully-factorized output distribution with a lightweight, tractable probabilistic inference layer. This formulation yields a distribution family that is significantly more expressive than standard factorized priors, enabling the modeling of complex joint dependencies, yet remains compact enough to avoid the prohibitive parameter explosion associated with full joint modeling. Empirically, CoDD seamlessly enhances diverse diffusion language model architectures with negligible overhead, matching the reasoning performance of computationally intensive Reinforcement Learning baselines at a fraction of the training cost. Furthermore, it prevents performance collapse in few-step generation, enabling high-quality outputs at significantly reduced latencies. Code available at: https://github.com/liuanji/CoDD