Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs

TL;DR

This paper introduces VCPO, a variance-controlled off-policy reinforcement learning method that stabilizes training of large language models in asynchronous settings by dynamically adjusting learning rates based on effective sample size, leading to improved stability and speed.

Contribution

The paper proposes VCPO, a novel off-policy RL algorithm that reduces variance through dynamic learning rate scaling and a closed-form baseline, enhancing asynchronous LLM training stability.

Findings

VCPO improves training stability across math and reasoning benchmarks.

It enables highly off-policy training with minimal variance issues.

Achieves 2.5x faster training in a tool-use task.

Abstract

Asynchronous reinforcement learning has become increasingly central to scaling LLM post-training, delivering major throughput gains by decoupling rollout generation from policy updates. However, widely used policy-gradient objectives such as REINFORCE and GRPO suffer under high asynchrony: stale rollouts produce heavy-tailed importance weights, so a small number of trajectories dominate updates and the policy-gradient estimator becomes markedly higher variance. Through systematic analysis on math, reasoning, and tool-use benchmarks, we find that this increasing variance is reliably predicted by collapsing effective sample size (ESS), which prior stabilization methods largely fail to address. Motivated by this diagnosis, we introduce $\textbf{V}$ariance $\textbf{C}$ontrolled $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{VCPO}$), a method that (i) dynamically scales the learning rate with ESS to dampen unreliable updates and (ii) applies a closed-form minimum-variance baseline for off-policy settings, without a critic model and adding minimal overhead. Empirically, across math and general reasoning benchmarks, this enables robustly stable asynchronous training compared to previous stabilization and algorithmic methods, even in highly off-policy regimes (128 steps off-policy). In a long-horizon, tool-use task, VCPO matches synchronous performance while delivering a 2.5$\times$ speedup in training time. Code is available at: https://github.com/mit-han-lab/vcpo