$V_{0.5}$: Generalist Value Model as a Prior for Sparse RL Rollouts

TL;DR

This paper introduces $V_{0.5}$, a novel value baseline for sparse reinforcement learning that adaptively combines a pre-trained value model with empirical data, improving stability and efficiency in policy gradient estimation.

Contribution

The paper proposes $V_{0.5}$, a new adaptive value baseline that fuses prior knowledge with sparse rollout data, using real-time statistical testing to optimize policy gradient stability.

Findings

$V_{0.5}$ outperforms GRPO and DAPO in six benchmarks.

Achieves faster convergence and over 10% performance gains.

Maintains low variance and stable gradients under extreme sparsity.

Abstract

In Reinforcement Learning with Verifiable Rewards (RLVR), constructing a robust advantage baseline is critical for policy gradients, effectively guiding the policy model to reinforce desired behaviors. Recent research has introduced Generalist Value Models (such as $V_0$), which achieve pre-trained value estimation by explicitly encoding model capabilities in-context, eliminating the need to synchronously update the value model alongside the policy model. In this paper, we propose $V_{0.5}$, which adaptively fuses the baseline predicted by such value model (acting as a prior) with the empirical mean derived from sparse rollouts. This constructs a robust baseline that balances computational efficiency with extremely low variance. Specifically, we introduce a real-time statistical testing and dynamic budget allocation. This balances the high variance caused by sparse sampling against the systematic bias (or hallucinations) inherent in the value model's prior. By constructing a hypothesis test to evaluate the prior's reliability in real-time, the system dynamically allocates additional rollout budget on demand. This mechanism minimizes the baseline estimator's Mean Squared Error (MSE), guaranteeing stable policy gradients, even under extreme sparsity with a group size of 4. Extensive evaluations across six mathematical reasoning benchmarks demonstrate that $V_{0.5}$ significantly outperforms GRPO and DAPO, achieving faster convergence and over some 10% performance improvement.