Accelerated Test-Time Scaling with Model-Free Speculative Sampling

TL;DR

STAND is a model-free speculative decoding method that accelerates language model reasoning by exploiting redundancy in reasoning paths, reducing inference latency by over 60% without accuracy loss.

Contribution

Introducing STAND, a novel model-free speculative decoding approach that leverages reasoning path redundancy for efficient, accurate, and versatile inference acceleration.

Findings

Reduces inference latency by 60-65% across multiple models and tasks.

Maintains accuracy comparable to standard decoding methods.

Outperforms existing speculative decoding techniques in diverse inference scenarios.

Abstract

Language models have demonstrated remarkable capabilities in reasoning tasks through test-time scaling techniques like best-of-N sampling and tree search. However, these approaches often demand substantial computational resources, creating a critical trade-off between performance and efficiency. We introduce STAND (STochastic Adaptive N-gram Drafting), a novel model-free speculative decoding approach that exploits the inherent redundancy in reasoning trajectories to achieve significant acceleration without compromising accuracy. Our analysis shows that reasoning paths frequently reuse similar reasoning patterns, enabling efficient model-free token prediction without requiring separate draft models. By introducing stochastic drafting and preserving probabilistic information through a memory-efficient logit-based N-gram module, combined with optimized Gumbel-Top-K sampling and data-driven tree construction, STAND significantly improves token acceptance rates. Extensive evaluations across multiple models and reasoning tasks (AIME-2024, GPQA-Diamond, and LiveCodeBench) demonstrate that STAND reduces inference latency by 60-65% compared to standard autoregressive decoding while maintaining accuracy. Furthermore, STAND consistently outperforms state-of-the-art speculative decoding methods across diverse inference patterns, including single-trajectory decoding, batch decoding, and test-time tree search. As a model-free approach, STAND can be applied to any existing language model without additional training, making it a powerful plug-and-play solution for accelerating language model reasoning.