DART: Diffusion-Inspired Speculative Decoding for Fast LLM Inference

TL;DR

DART introduces a diffusion-inspired speculative decoding method that predicts multiple future tokens in parallel, significantly accelerating large language model inference while maintaining high accuracy.

Contribution

It proposes a novel parallel logit prediction approach and an efficient tree pruning algorithm, reducing decoding latency and outperforming existing methods like EAGLE3.

Findings

Achieves 2.03x to 3.44x speedup in decoding time.

Surpasses EAGLE3 by 30% on average in speed.

Maintains high draft accuracy with reduced overhead.

Abstract

Speculative decoding is an effective and lossless approach for accelerating LLM inference. However, existing widely adopted model-based draft designs, such as EAGLE3, improve accuracy at the cost of multi-step autoregressive inference, resulting in high drafting latency and ultimately rendering the drafting stage itself a performance bottleneck. Inspired by diffusion-based large language models (dLLMs), we propose DART, which leverages parallel generation to reduce drafting latency. DART predicts logits for multiple future masked positions in parallel within a single forward pass based on hidden states of the target model, thereby eliminating autoregressive rollouts in the draft model while preserving a lightweight design. Based on these parallel logit predictions, we further introduce an efficient tree pruning algorithm that constructs high-quality draft token trees with N-gram-enforced semantic continuity. DART substantially reduces draft-stage overhead while preserving high draft accuracy, leading to significantly improved end-to-end decoding speed. Experimental results demonstrate that DART achieves a 2.03x--3.44x wall-clock time speedup across multiple datasets, surpassing EAGLE3 by 30% on average and offering a practical speculative decoding framework. Code is released at https://github.com/fvliang/DART.