WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference

TL;DR

WeDLM introduces a diffusion decoding framework using standard causal attention, enabling faster parallel inference in large language models while maintaining quality, outperforming optimized autoregressive engines in speed.

Contribution

The paper presents WeDLM, a novel diffusion decoding method that uses causal attention and topological reordering to achieve efficient, parallel, and high-quality language model inference.

Findings

Approaches 3x speedup on reasoning benchmarks.

Achieves up to 10x speedup in low-entropy generation.

Outperforms optimized autoregressive engines in deployment settings.

Abstract

Autoregressive (AR) generation is the standard decoding paradigm for Large Language Models (LLMs), but its token-by-token nature limits parallelism at inference time. Diffusion Language Models (DLLMs) offer parallel decoding by recovering multiple masked tokens per step; however, in practice they often fail to translate this parallelism into deployment speed gains over optimized AR engines (e.g., vLLM). A key reason is that many DLLMs rely on bidirectional attention, which breaks standard prefix KV caching and forces repeated contextualization, undermining efficiency. We propose WeDLM, a diffusion decoding framework built entirely on standard causal attention to make parallel generation prefix-cache friendly. The core idea is to let each masked position condition on all currently observed tokens while keeping a strict causal mask, achieved by Topological Reordering that moves observed tokens to the physical prefix while preserving their logical positions. Building on this property, we introduce a streaming decoding procedure that continuously commits confident tokens into a growing left-to-right prefix and maintains a fixed parallel workload, avoiding the stop-and-wait behavior common in block diffusion methods. Experiments show that WeDLM preserves the quality of strong AR backbones while delivering substantial speedups, approaching 3x on challenging reasoning benchmarks and up to 10x in low-entropy generation regimes; critically, our comparisons are against AR baselines served by vLLM under matched deployment settings, demonstrating that diffusion-style decoding can outperform an optimized AR engine in practice.