Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards

TL;DR

This paper introduces VIP, a variance-informed adaptive rollout allocation method for online reinforcement learning with verifiable rewards, improving sampling efficiency and training performance.

Contribution

We propose VIP, a novel adaptive allocation strategy using Gaussian process predictions to minimize gradient variance, enhancing efficiency over fixed or heuristic methods.

Findings

VIP outperforms uniform allocation in benchmarks.

VIP achieves higher policy performance with fewer rollouts.

The method effectively reduces gradient variance during training.

Abstract

Sampling efficiency is a key bottleneck in reinforcement learning with verifiable rewards. Existing group-based policy optimization methods, such as GRPO, allocate a fixed number of rollouts for all training prompts. This uniform allocation implicitly treats all prompts as equally informative, and could lead to inefficient computational budget usage and impede training progress. We introduce VIP, a Variance-Informed Predictive allocation strategy that allocates a given rollout budget to the prompts in the incumbent batch to minimize the expected gradient variance of the policy update. At each iteration, VIP uses a lightweight Gaussian process model to predict per-prompt success probabilities based on recent rollouts. These probability predictions are translated into variance estimates, which are then fed into a convex optimization problem to determine the optimal rollout allocations under a hard compute budget constraint. Empirical results show that VIP consistently improves sampling efficiency and achieves higher performance than uniform or heuristic allocation strategies in multiple benchmarks.