DepCap: Adaptive Block-Wise Parallel Decoding for Efficient Diffusion LM Inference

TL;DR

DepCap introduces an adaptive, training-free framework for block-wise diffusion language model inference, improving speed without sacrificing quality by using influence signals for dynamic block partitioning and conflict detection.

Contribution

It proposes a novel adaptive, training-free method for block-wise DLM inference that enhances speed-quality trade-offs through influence-based block extension and conflict-aware parallel decoding.

Findings

Achieves up to 5.63× speedup without significant quality loss.

Applicable to various DLM architectures and compatible with existing KV-cache strategies.

Demonstrates improved inference efficiency across multiple benchmarks.

Abstract

Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive language generation due to their potential for parallel decoding and global refinement of the entire sequence. To unlock this potential, DLM inference must carefully balance generation quality and decoding speed. Recent block-wise DLM decoding methods improve this trade-off by performing diffusion-based decoding sequentially in blocks. However, existing methods typically rely on fixed block schedules or current-step local signals to determine block boundaries, and use conservative confidence-based parallel decoding to avoid conflicts, limiting the quality-speed trade-off. In this paper, we argue that block-wise DLM inference requires more suitable signals for its two core decisions: cross-step signals for determining block boundaries, and token-level conflict signals for parallel decoding. Based on this view, we propose DepCap, a training-free framework for efficient block-wise DLM inference. Specifically, DepCap instantiates the cross-step signal as the influence of the last decoded block and uses it to adaptively determine how far the next block should extend, while identifying a conflict-free subset of tokens for safe parallel decoding within each block, enabling substantial inference acceleration with negligible quality degradation. DepCap is a plug-and-play method applicable to various DLMs, and compatible with existing KV-cache strategies for block-wise DLM. An information-theoretic analysis further suggests that the cumulative last-block influence on a candidate block is approximately additive across tokens, supporting the proposed block-partitioning criterion. Experimental results show that DepCap achieves favorable speed-quality trade-offs across multiple DLM backbones and reasoning and coding benchmarks, with up to 5.63$\times$ speedup without significant performance degradation.