Scaling Speculative Decoding with Lookahead Reasoning

TL;DR

Lookahead Reasoning enhances speculative decoding by introducing a step-level parallelism layer, significantly increasing decoding speed while maintaining answer quality across various benchmarks.

Contribution

This paper introduces Lookahead Reasoning, a novel method that combines step-level and token-level parallelism to surpass existing speculative decoding speed limits.

Findings

Speeds up speculative decoding from 1.4x to 2.1x.

Maintains answer quality across multiple benchmarks.

Scales better with additional GPU throughput.

Abstract

Reasoning models excel by generating long chain-of-thoughts, but decoding the resulting thousands of tokens is slow. Token-level speculative decoding (SD) helps, but its benefit is capped, because the chance that an entire $\gamma$-token guess is correct falls exponentially as $\gamma$ grows. This means allocating more compute for longer token drafts faces an algorithmic ceiling -- making the speedup modest and hardware-agnostic. We raise this ceiling with Lookahead Reasoning, which exploits a second, step-level layer of parallelism. Our key insight is that reasoning models generate step-by-step, and each step needs only to be semantically correct, not exact token matching. In Lookahead Reasoning, a lightweight draft model proposes several future steps; the target model expands each proposal in one batched pass, and a verifier keeps semantically correct steps while letting the target regenerate any that fail. Token-level SD still operates within each reasoning step, so the two layers of parallelism multiply. We show Lookahead Reasoning lifts the peak speedup of SD both theoretically and empirically. Across GSM8K, AIME, and other benchmarks, Lookahead Reasoning improves the speedup of SD from 1.4x to 2.1x while preserving answer quality, and its speedup scales better with additional GPU throughput. Our code is available at https://github.com/hao-ai-lab/LookaheadReasoning