TALON: Confidence-Aware Speculative Decoding with Adaptive Token Trees

TL;DR

TALON introduces an adaptive, budget-driven speculative decoding framework that dynamically adjusts the draft tree structure, significantly accelerating large language model inference without quality loss.

Contribution

It proposes TALON, a training-free, adaptive tree expansion method that optimizes speculative decoding by dynamically shaping the draft tree based on token difficulty and context.

Findings

Achieves up to 5.16x speedup over autoregressive decoding.

Outperforms state-of-the-art EAGLE-3 across multiple models and datasets.

Effectively balances exploration and depth in tree-based decoding.

Abstract

Speculative decoding (SD) has become a standard technique for accelerating LLM inference without sacrificing output quality. Recent advances in speculative decoding have shifted from sequential chain-based drafting to tree-structured generation, where the draft model constructs a tree of candidate tokens to explore multiple possible drafts in parallel. However, existing tree-based SD methods typically build a fixed-width, fixed-depth draft tree, which fails to adapt to the varying difficulty of tokens and contexts. As a result, the draft model cannot dynamically adjust the tree structure to early stop on difficult tokens and extend generation for simple ones. To address these challenges, we introduce TALON, a training-free, budget-driven adaptive tree expansion framework that can be plugged into existing tree-based methods. Unlike static methods, TALON constructs the draft tree iteratively until a fixed token budget is met, using a hybrid expansion strategy that adaptively allocates the node budget to each layer of the draft tree. This framework naturally shapes the draft tree into a "deep-and-narrow" form for deterministic contexts and a "shallow-and-wide" form for uncertain branches, effectively optimizing the trade-off between exploration width and generation depth under a given budget. Extensive experiments across 5 models and 6 datasets demonstrate that TALON consistently outperforms state-of-the-art EAGLE-3, achieving up to 5.16x end-to-end speedup over auto-regressive decoding.