SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection

TL;DR

SpecKV introduces an adaptive speculation length controller for speculative decoding in LLMs, optimizing token proposal per step based on draft model signals and compression levels, significantly improving efficiency.

Contribution

We develop a lightweight adaptive controller that dynamically selects speculation length using draft model signals, tailored to compression regimes, enhancing decoding efficiency.

Findings

SpecKV achieves 56.0% improvement over fixed speculation length baseline.

Draft model confidence and entropy strongly predict acceptance rates.

Adaptive speculation length varies across compression levels, optimizing token proposal.

Abstract

Speculative decoding accelerates large language model (LLM) inference by using a small draft model to propose candidate tokens that a larger target model verifies. A critical hyperparameter in this process is the speculation length $\gamma$, which determines how many tokens the draft model proposes per step. Nearly all existing systems use a fixed $\gamma$ (typically 4), yet empirical evidence suggests that the optimal value varies across task types and, crucially, depends on the compression level applied to the target model. In this paper, we present SpecKV, a lightweight adaptive controller that selects $\gamma$ per speculation step using signals extracted from the draft model itself. We profile speculative decoding across 4 task categories, 4 speculation lengths, and 3 compression levels (FP16, INT8, NF4), collecting 5,112 step-level records with per-step acceptance rates, draft entropy, and draft confidence. We demonstrate that the optimal $\gamma$ shifts across compression regimes and that draft model confidence and entropy are strong predictors of acceptance rate (correlation $\approx$ 0.56). SpecKV uses a small MLP trained on these signals to maximize expected tokens per speculation step, achieving a 56.0% improvement over the fixed-$\gamma=4$ baseline with only 0.34 ms overhead per decision (<0.5% of step time). The improvement is statistically significant (p < 0.001, paired bootstrap test). We release all profiling data, trained models, and notebooks as open-source artifacts.