DAWN: Dependency-Aware Fast Inference for Diffusion LLMs

TL;DR

DAWN is a dependency-aware decoding method that significantly accelerates diffusion LLM inference by intelligently selecting unmasking positions based on token dependencies, maintaining high quality while improving speed.

Contribution

DAWN introduces a training-free, dependency-aware decoding strategy that models token dependencies to enhance parallel inference efficiency for diffusion LLMs.

Findings

Speeds up inference by 1.80-8.06x across models and datasets.

Maintains comparable generation quality with baseline methods.

Effectively leverages token dependency graphs for improved parallel decoding.

Abstract

Diffusion large language models (dLLMs) have shown advantages in text generation, particularly due to their inherent ability for parallel decoding. However, constrained by the quality--speed trade-off, existing inference solutions adopt conservative parallel strategies, leaving substantial efficiency potential underexplored. A core challenge is that parallel decoding assumes each position can be filled independently, but tokens are often semantically coupled. Thus, the correct choice at one position constrains valid choices at others. Without modeling these inter-token dependencies, parallel strategies produce deteriorated outputs. Motivated by this insight, we propose DAWN, a training-free, dependency-aware decoding method for fast dLLM inference. DAWN extracts token dependencies and leverages two key motivations: (1) positions dependent on unmasked certain positions become more reliable, (2) simultaneously unmasking strongly coupled uncertain positions induces errors. Given those findings, DAWN leverages a dependency graph to select more reliable unmasking positions at each iteration, achieving high parallelism with negligible loss in generation quality. Extensive experiments across multiple models and datasets demonstrate that DAWN speedups the inference by 1.80-8.06x over baselines while preserving the generation quality. Code is released at https://github.com/lizhuo-luo/DAWN.