Arbitrage: Efficient Reasoning via Advantage-Aware Speculation

TL;DR

Arbitrage introduces a dynamic routing framework for step-level speculative decoding in large language models, significantly improving inference efficiency by selectively choosing when to generate with draft or target models, especially in reasoning tasks.

Contribution

It proposes Arbitrage, a novel step-level speculative generation method that uses a lightweight router to dynamically select the best model for each step, enhancing efficiency and accuracy in reasoning tasks.

Findings

Reduces inference latency by up to ~2x at matched accuracy.

Outperforms prior step-level speculative decoding methods.

Achieves near-optimal efficiency-accuracy trade-offs across benchmarks.

Abstract

Modern Large Language Models achieve impressive reasoning capabilities with long Chain of Thoughts, but they incur substantial computational cost during inference, and this motivates techniques to improve the performance-cost ratio. Among these techniques, Speculative Decoding accelerates inference by employing a fast but inaccurate draft model to autoregressively propose tokens, which are then verified in parallel by a more capable target model. However, due to unnecessary rejections caused by token mismatches in semantically equivalent steps, traditional token-level Speculative Decoding struggles in reasoning tasks. Although recent works have shifted to step-level semantic verification, which improve efficiency by accepting or rejecting entire reasoning steps, existing step-level methods still regenerate many rejected steps with little improvement, wasting valuable target compute. To address this challenge, we propose Arbitrage, a novel step-level speculative generation framework that routes generation dynamically based on the relative advantage between draft and target models. Instead of applying a fixed acceptance threshold, Arbitrage uses a lightweight router trained to predict when the target model is likely to produce a meaningfully better step. This routing approximates an ideal Arbitrage Oracle that always chooses the higher-quality step, achieving near-optimal efficiency-accuracy trade-offs. Across multiple mathematical reasoning benchmarks, Arbitrage consistently surpasses prior step-level Speculative Decoding baselines, reducing inference latency by up to $\sim2\times$ at matched accuracy.