Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match

TL;DR

FLy is a training-free decoding method that accepts semantically correct drafts beyond exact matches, significantly speeding up large language model inference while maintaining high accuracy, especially on out-of-domain data.

Contribution

FLy introduces a training-free, loosely speculative decoding approach with a two-tier verification mechanism and multi-level acceleration, improving speed and robustness across models and domains.

Findings

Preserves over 99% of target accuracy.

Achieves 2.81x speedup on Llama-3.1-70B-Instruct.

Outperforms training-based EAGLE-3 by 1.62x on OOD datasets.

Abstract

Large language models (LLMs) achieve strong performance across diverse tasks but suffer from high inference latency due to their autoregressive generation. Speculative Decoding (SPD) mitigates this issue by verifying candidate tokens in parallel from a smaller draft model, yet its strict exact-match verification discards many semantically valid continuations. Moreover, existing training-based SPD methods often suffer from performance degradation on out-of-distribution (OOD) tasks. To this end, we propose Training-Free Loosely Speculative Decoding (FLy), a novel method that loosens the rigid verification criterion by leveraging the target model's self-corrective behavior to judge whether a draft-target mismatch remains semantically valid. FLy introduces a two-tier mechanism: an entropy-level gate that identifies whether the current token allows multiple plausible alternatives or is nearly deterministic, and a token-level deferred window that distinguishes genuine errors from differently worded yet semantically correct variants. To further reduce latency, we design a multi-level acceleration strategy that accelerates not only the target model but also the drafter itself. Owing to its training-free design, FLy composes seamlessly with arbitrary draft-target pairs and generalizes across models and domains without hyperparameter re-tuning. Experiments show that FLy preserves more than 99% of the target model's accuracy while achieving an average 2.81x speedup on Llama-3.1-70B-Instruct and 5.07x speedup on the 405B variant. Notably, on out-of-domain datasets, our method remains highly effective and outperforms the training-based method EAGLE-3 by 1.62x.