SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

TL;DR

This paper introduces SpecBound, a self-draft decoding framework for LLMs that enhances speed by adaptive speculation control and confidence calibration, maintaining accuracy without modifying the base model.

Contribution

It proposes a novel self-draft method with layer-wise confidence calibration and adaptive speculation bounds, significantly improving decoding speed while preserving output correctness.

Findings

Achieves up to 2.33x speedup over standard decoding.

Effectively suppresses overconfidence in early layers.

Maintains exact output equivalence with the original model.

Abstract

Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models (LLMs). Self-draft methods, which leverage the base LLM itself for speculation, avoid the overhead of auxiliary draft models but face limitations: shallow layers often produce overconfident yet incorrect token predictions, and the presence of difficult tokens in a draft sequence forces redundant computation through deeper layers, undermining both draft acceptance and overall speedup. To address these issues, we propose a novel self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision and adaptively bounds speculation length based on token-wise decoding difficulty. By reprocessing the hidden states of draft tokens in a unified parallel pass through deep layers, our method maintains exact output equivalence with the original model while maximizing computational efficiency. It requires no modifications to the base LLM parameters and achieves up to 2.33x wall-time speedup over standard autoregressive decoding across diverse long-form generation tasks and multiple model architectures.