From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning

TL;DR

SpecGuard introduces a verification-aware speculative decoding method that uses internal model signals for step-level validation, enhancing accuracy and efficiency in multi-step reasoning tasks.

Contribution

It proposes a novel internal signal-based verification framework for speculative decoding, reducing reliance on external reward models and improving inference performance.

Findings

Improves reasoning accuracy by 3.6%

Reduces latency by approximately 11%

Outperforms existing speculative decoding methods

Abstract

Speculative decoding (SD) accelerates large language model inference by allowing a lightweight draft model to propose outputs that a stronger target model verifies. However, its token-centric nature allows erroneous steps to propagate. Prior approaches mitigate this using external reward models, but incur additional latency, computational overhead, and limit generalizability. We propose SpecGuard, a verification-aware speculative decoding framework that performs step-level verification using only model-internal signals. At each step, SpecGuard samples multiple draft candidates and selects the most consistent step, which is then validated using an ensemble of two lightweight model-internal signals: (i) an attention-based grounding score that measures attribution to the input and previously accepted steps, and (ii) a log-probability-based score that captures token-level confidence. These signals jointly determine whether a step is accepted or recomputed using the target, allocating compute selectively. Experiments across a range of reasoning benchmarks show that SpecGuard improves accuracy by 3.6% while reducing latency by ~11%, outperforming both SD and reward-guided SD.