Stabilizing Reinforcement Learning with LLMs: Formulation and Practices

TL;DR

This paper introduces a new formulation for reinforcement learning with large language models, explaining the conditions under which surrogate objectives effectively optimize sequence-level rewards, and provides practical stabilization techniques validated through extensive experiments.

Contribution

It offers a theoretical framework for understanding RL stability with LLMs and develops practical training recipes validated on large-scale models.

Findings

Importance sampling correction improves training stability.

Clipping and Routing Replay are crucial for off-policy stability.

Stable training leads to consistent final performance regardless of initialization.

Abstract

This paper proposes a novel formulation for reinforcement learning (RL) with large language models, explaining why and under what conditions the true sequence-level reward can be optimized via a surrogate token-level objective in policy gradient methods such as REINFORCE. Specifically, through a first-order approximation, we show that this surrogate becomes increasingly valid only when both the training-inference discrepancy and policy staleness are minimized. This insight provides a principled explanation for the crucial role of several widely adopted techniques in stabilizing RL training, including importance sampling correction, clipping, and particularly Routing Replay for Mixture-of-Experts (MoE) models. Through extensive experiments with a 30B MoE model totaling hundreds of thousands of GPU hours, we show that for on-policy training, the basic policy gradient algorithm with importance sampling correction achieves the highest training stability. When off-policy updates are introduced to accelerate convergence, combining clipping and Routing Replay becomes essential to mitigate the instability caused by policy staleness. Notably, once training is stabilized, prolonged optimization consistently yields comparable final performance regardless of cold-start initialization. We hope that the shared insights and the developed recipes for stable RL training will facilitate future research.