Beat the long tail: Distribution-Aware Speculative Decoding for RL Training

TL;DR

This paper introduces DAS, a distribution-aware speculative decoding method that significantly accelerates reinforcement learning rollouts for large language models by leveraging rollout history and adaptive drafting, reducing time by up to 50%.

Contribution

DAS is a novel framework that uses rollout history and a length-aware policy to speed up RL training without changing model outputs.

Findings

Reduces rollout time by up to 50%.

Preserves training curves and model performance.

Effective on math and code reasoning tasks.

Abstract

Reinforcement learning(RL) post-training has become essential for aligning large language models (LLMs), yet its efficiency is increasingly constrained by the rollout phase, where long trajectories are generated token by token. We identify a major bottleneck:the long-tail distribution of rollout lengths, where a small fraction of long generations dominates wall clock time and a complementary opportunity; the availability of historical rollouts that reveal stable prompt level patterns across training epochs. Motivated by these observations, we propose DAS, a Distribution Aware Speculative decoding framework that accelerates RL rollouts without altering model outputs. DAS integrates two key ideas: an adaptive, nonparametric drafter built from recent rollouts using an incrementally maintained suffix tree, and a length aware speculation policy that allocates more aggressive draft budgets to long trajectories that dominate makespan. This design exploits rollout history to sustain acceptance while balancing base and token level costs during decoding. Experiments on math and code reasoning tasks show that DAS reduces rollout time up to 50% while preserving identical training curves, demonstrating that distribution-aware speculative decoding can significantly accelerate RL post training without compromising learning quality.