Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients

TL;DR

This paper introduces Positive-Only Policy Optimization (POPO), a reinforcement learning framework that learns solely from positive rollouts, achieving competitive or superior performance to existing methods like GRPO in language model benchmarks.

Contribution

POPO is a novel RLVR framework that eliminates the need for negative rollouts, using implicit negative gradients and stabilization techniques to improve policy optimization.

Findings

POPO achieves 36.67% in AIME 2025 with Qwen-Math-7B, outperforming GRPO's 30.00%.

POPO's components are shown to be necessary and robust through ablation studies.

Experiments demonstrate POPO's performance is comparable or superior to GRPO across mathematical benchmarks.

Abstract

Reinforcement learning with verifiable rewards (RLVR), due to the deterministic verification, becomes a dominant paradigm for enhancing the reasoning ability of large language models (LLMs). The community witnesses the rapid change from the Proximal Policy Optimization (PPO) to Group Relative Policy Optimization (GRPO), in which GRPO reduces the complicated advantage estimation with simple estimation over grouped positive and negative rollouts. However, we note that negative rollouts may admit no gradation of failure severity, and the combinatorial vastness makes penalizing a few sampled negatives unlikely to cover a meaningful reward signal under sparse binary rewards. In this work, we propose Positive-Only Policy Optimization (POPO), a novel RLVR framework in which learning can occur exclusively via online positive rollouts. Specifically, POPO utilizes bounded importance sampling over the positive rollout set. Thus, no disjoint negative rollouts are used for the gradient guidance. We show that implicit negative gradients can emerge naturally through reinforcing the positive probability via rollouts redistribution. Next, POPO stabilizes the policy optimization through two mechanisms. First, it applies a siamese policy network with a momentum-based adaptation law for stabilized policy evolution. Second, we replace the KL-divergence with a bounded similarity penalty term in the siamese representation space. We conduct extensive experiments using publicly available, well-established text-LLM models, e.g., the Qwen family, across all-level mathematical benchmarks. Our experiment demonstrates that POPO achieves performance comparable to, or even superior to GRPO. Notably, we show that POPO can achieve 36.67% in AIME 2025 with Qwen-Math-7B, outperforming GRPO 30.00%. Our ablation and sweep studies further illustrate the necessity and robustness of POPO components.