Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference

TL;DR

Calibrated Speculative Decoding (CSD) is a training-free framework that improves inference speed and accuracy in large language models by intelligently recovering discarded tokens using frequency guidance and probability checks.

Contribution

CSD introduces a novel, training-free approach with lightweight modules for better token recovery, outperforming existing methods in speed and accuracy across diverse models.

Findings

Achieves up to 2.33x throughput speedup.

Maintains model accuracy across various tasks.

Enhances performance on complex reasoning datasets.

Abstract

Speculative decoding accelerates autoregressive generation by letting draft tokens bypass full verification, but conventional frameworks suffer from frequent false rejections, particularly when draft models produce semantically correct but lexically divergent outputs. In this paper, we present Calibrated Speculative Decoding (CSD), a training-free framework that recovers valid tokens discarded by standard verification. Guided by the principle of "Frequency-Guided Candidate Selection and Probability-Guarded Acceptance," CSD incorporates two lightweight modules: Online Correction Memory, which aggregates historical rejections to propose recurring divergence patterns as rescue candidates, and Semantic Consistency Gating, which verifies candidate admissibility using probability ratios instead of exact token matching. Our evaluation across diverse large language models demonstrates that CSD outperforms existing methods, achieving a peak throughput speedup of 2.33x. CSD preserves model accuracy across all tasks while further boosting performance on complex reasoning datasets. These results establish CSD as a highly effective, lightweight solution for practical LLM deployments.