SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting

TL;DR

SpecBlock introduces a novel block-iterative speculative decoding method that combines path dependence with efficient drafting, significantly improving LLM inference speed and adaptability.

Contribution

It proposes a new block-iterative drafter with mechanisms for path dependence, dynamic tree growth, and cost-aware adaptation, enhancing speed and efficiency over existing methods.

Findings

Achieves 8-13% mean speedup over EAGLE-3 at 44-52% of its drafting cost.

Cost-aware adaptation extends speedup to 11-19%.

Improves inference efficiency by combining path dependence with dynamic tree drafting.

Abstract

Speculative decoding accelerates LLM inference by drafting a tree of candidate continuations and verifying it in one target forward. Existing drafters fall into two camps with opposite weaknesses. Autoregressive drafters such as EAGLE-3 preserve dependence along each draft path but call the drafter once per tree depth, making drafting a non-trivial share of per-iteration latency. Parallel drafters cut drafter calls by predicting multiple future positions in one forward, but each position is predicted without seeing the others, producing paths the verifier rejects. In this paper, we propose SpecBlock, a block-iterative drafter that combines path dependence with cheap drafting. Each drafter forward produces K dependent positions and we call this a block. The draft tree grows through repeated block expansions. Two mechanisms explicitly carry path dependence to keep later draft positions accurate. Within each block, a layer-wise shift carries the previous position's hidden state into every decoder layer. Across blocks, each new block can start from any position of the previous block, inheriting its hidden state to extend the path. To spend verifier budget where acceptance is likely, a co-trained rank head replaces the fixed top-k tree by allocating per-position branching during drafting. To avoid training the drafter on prefixes it never produces at inference, a valid-prefix mask drops the loss at later positions once an earlier one is wrong. Beyond static drafting, a cost-aware bandit at deployment uses free verifier feedback to update the drafter selectively, only when the expected throughput gain exceeds the update cost. Experiments show that SpecBlock improves mean speedup by 8-13% over EAGLE-3 at 44-52% of its drafting cost, and cost-aware adaptation extends this lead to 11-19%.