How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization

TL;DR

DynaMO introduces a theoretically grounded framework for resource allocation and advantage modulation in reinforcement learning, improving policy optimization for language models by addressing gradient variance and attenuation issues.

Contribution

It derives variance-minimizing allocation strategies and gradient-aware advantage modulation, enhancing RLVR performance on reasoning benchmarks.

Findings

Consistent improvements over strong RLVR baselines on reasoning tasks.

Variance-minimizing allocation reduces gradient variance effectively.

Advantage modulation stabilizes training by compensating for gradient attenuation.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for Large Language Model (LLM) reasoning, yet current methods face key challenges in resource allocation and policy optimization dynamics: (i) uniform rollout allocation ignores gradient variance heterogeneity across problems, and (ii) the softmax policy structure causes gradient attenuation for high-confidence correct actions, while excessive gradient updates may destabilize training. Therefore, we propose DynaMO, a theoretically-grounded dual-pronged optimization framework. At the sequence level, we prove that uniform allocation is suboptimal and derive variance-minimizing allocation from the first principle, establishing Bernoulli variance as a computable proxy for gradient informativeness. At the token level, we develop gradient-aware advantage modulation grounded in theoretical analysis of gradient magnitude bounds. Our framework compensates for gradient attenuation of high-confidence correct actions while utilizing entropy changes as computable indicators to stabilize excessive update magnitudes. Extensive experiments conducted on a diverse range of mathematical reasoning benchmarks demonstrate consistent improvements over strong RLVR baselines. Our implementation is available at: https://github.com/GithubX-F/DynaMO-RL.