Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States

TL;DR

SemanticSpec enhances speculative decoding for large language models by verifying entire semantic sequences, significantly improving inference speed while maintaining accuracy across multiple benchmarks.

Contribution

It introduces a novel semantic-aware decoding framework that probes internal states to verify meaning, surpassing token-level methods in efficiency and effectiveness.

Findings

Achieves up to 2.7x speedup on DeepSeekR1-32B

Achieves up to 2.1x speedup on QwQ-32B

Outperforms existing token-level and sequence-level baselines

Abstract

Large Language Models (LLMs) achieve strong performance across many tasks but suffer from high inference latency due to autoregressive decoding. The issue is exacerbated in Large Reasoning Models (LRMs), which generate lengthy chains of thought. While speculative decoding accelerates inference by drafting and verifying multiple tokens in parallel, existing methods operate at the token level and ignore semantic equivalence (i.e., different token sequences expressing the same meaning), leading to inefficient rejections. We propose SemanticSpec, a semantic-aware speculative decoding framework that verifies entire semantic sequences instead of tokens. SemanticSpec introduces a semantic probability estimation mechanism that probes the model's internal hidden states to assess the likelihood of generating sequences with specific meanings. Experiments on four benchmarks show that SemanticSpec achieves up to 2.7x speedup on DeepSeekR1-32B and 2.1x on QwQ-32B, consistently outperforming token-level and sequence-level baselines in both efficiency and effectiveness.