ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systems

TL;DR

ReSpec is a system that optimizes speculative decoding in reinforcement learning for large language models, significantly speeding up training without sacrificing reward quality or stability.

Contribution

ReSpec introduces dynamic tuning, knowledge distillation, and reward-weighted updates to effectively integrate speculative decoding into RL training of LLMs.

Findings

Achieves up to 4.5x training speedup on Qwen models

Maintains reward convergence and training stability

Addresses key challenges of speculative decoding in RL systems

Abstract

Adapting large language models (LLMs) via reinforcement learning (RL) is often bottlenecked by the generation stage, which can consume over 75\% of the training time. Speculative decoding (SD) accelerates autoregressive generation in serving systems, but its behavior under RL training remains largely unexplored. We identify three critical gaps that hinder the naive integration of SD into RL systems: diminishing speedups at large batch sizes, drafter staleness under continual actor updates, and drafter-induced policy degradation. To address these gaps, we present ReSpec, a system that adapts SD to RL through three complementary mechanisms: dynamically tuning SD configurations, evolving the drafter via knowledge distillation, and weighting updates by rollout rewards. On Qwen models (3B--14B), ReSpec achieves up to 4.5x speedup while preserving reward convergence and training stability, providing a practical solution for efficient RL-based LLM adaptation.