Consistent Diffusion Language Models

TL;DR

This paper introduces a novel training framework called Consistent Diffusion Language Model (CDLM) that improves discrete diffusion models for faster, high-quality text generation by ensuring path-invariance across stochastic bridges.

Contribution

It proposes Multi-Path Discrete Consistency (MPDC) and unifies various diffusion approaches into a single, scalable training method for discrete language models.

Findings

CDLM achieves state-of-the-art results in text generation tasks.

It outperforms existing discrete diffusion models across different sampling budgets.

Significant improvements are observed in low-step, fast sampling regimes.

Abstract

Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps. In continuous domains, consistency training along the probability-flow ODE is a popular recipe to accelerate diffusion. For discrete diffusion, no analogous sample-space ODE exists, making direct adaptation ill-defined. We argue that the natural discrete substitute is not a deterministic trajectory but its stochastic counterpart: the exact posterior bridge, available in closed form for broad corruption families including masked and uniform diffusion. Building on this observation, we introduce Multi-Path Discrete Consistency (MPDC), a new principle that trains a denoiser to be path-invariant in expectation across these stochastic bridges, and instantiate it as the Consistent Diffusion Language Model (CDLM), a single-stage, teacher-free training framework. A single CDLM objective unifies masked diffusion, continuous consistency models, and progressive/discrete distillation as analytic limits or empirical approximations of one common view. Empirically, CDLM establishes a new state of the art on both conditional and unconditional text-generation, consistently outperforming strong base discrete diffusion models and often even multi-stage distilled baselines across sampling budgets, with the largest gains in the few-step regime. Together, these results position CDLM as a principled and scalable foundation for the next generation of fast, high-fidelity discrete generative modeling.