$\texttt{SPECS}$: Faster Test-Time Scaling through Speculative Drafts

TL;DR

SPECS is a latency-aware test-time scaling method for large language models that uses speculative drafts and reward signals to improve efficiency and reduce latency without sacrificing accuracy.

Contribution

It introduces SPECS, a novel approach combining speculative decoding with reward-guided evaluation to optimize test-time scaling under latency constraints.

Findings

Reduces latency by up to 19.1% while maintaining or improving accuracy.

Matches or surpasses beam search performance on multiple datasets.

Converges to a KL-regularized reinforcement learning solution.

Abstract

Scaling test-time compute has driven the recent advances in the reasoning capabilities of large language models (LLMs), typically by allocating additional computation for more thorough exploration. However, increased compute often comes at the expense of higher user-facing latency, directly impacting user experience. Current test-time scaling methods primarily optimize for accuracy based on total compute resources (FLOPS), often overlooking latency constraints. To address this gap, we propose $\texttt{SPECS}$, a latency-aware test-time scaling method inspired by speculative decoding. $\texttt{SPECS}$~uses a smaller, faster model to generate candidate sequences efficiently, and evaluates these candidates using signals from both a larger target model and a dedicated reward model. We introduce new integration strategies, including reward-guided soft verification and a reward-based deferral mechanism. Empirical results on MATH500, AMC23 and OlympiadBench datasets show that $\texttt{SPECS}$~matches or surpasses beam search accuracy while reducing latency by up to $\sim$19.1\%. Our theoretical analysis shows that our algorithm converges to the solution of a KL-regularized reinforcement learning objective with increasing beam width.