Locally Coherent Parallel Decoding in Diffusion Language Models

TL;DR

This paper introduces CoDiLA, a method that combines parallel decoding with local dependency modeling in diffusion language models, significantly improving code generation coherence and speed.

Contribution

CoDiLA employs a small auxiliary autoregressive model to ensure local coherence during parallel decoding in diffusion language models, enhancing generation quality.

Findings

Eliminates coherence artifacts with a 0.6B parameter auxiliary AR model

Achieves a new Pareto frontier for accuracy and speed in code generation

Maintains core bidirectional capabilities of diffusion language models

Abstract

Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) models, offering sub-linear generation latency and bidirectional capabilities that are particularly appealing for code generation and editing. Achieving sub-linear latency in discrete DLMs requires predicting multiple tokens in parallel. However, standard DLMs sample tokens independently from conditional marginal distributions, failing to capture the joint dependencies among concurrently generated tokens. As a result, they often lead to syntactic inconsistencies and break multi-token structures. In this work, we introduce CoDiLA (Coherent Diffusion with Local Autoregression), a method that reconciles parallel sampling with local dependency modeling. Rather than forcing the DLM to resolve fine-grained syntax, CoDiLA delegates local decoding to a small, auxiliary AR model operating on the diffusion latents. This design allows for parallel generation while ensuring sequential validity within a block and maintaining core DLM capabilities, including bidirectional modeling across blocks. We demonstrate that using a highly compact auxiliary AR model (e.g., 0.6B parameters) effectively eliminates coherence artifacts, establishing a new Pareto frontier for accuracy and speed in code generation benchmarks.