Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR

TL;DR

This paper introduces AsymGRPO, a novel advantage modulation method for RLVR that selectively enhances productive entropy and suppresses noisy entropy, improving reasoning performance in large language models.

Contribution

It proposes a channel-wise advantage modulation approach that decouples positive and negative advantage updates, enabling more precise control over exploration and exploitation in RLVR.

Findings

AsymGRPO outperforms existing RLVR methods on five reasoning benchmarks.

Decoupling advantage channels improves model's reasoning accuracy.

Flexible modulation of advantage channels enhances learning across prompt difficulties.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning ability of large language models (LLMs), but it often suffers from \textit{restricted exploration}, where the policy rapidly concentrates on a narrow set of solutions. A common remedy is entropy regularization, which attempts to preserve exploration by increasing policy entropy. However, for LLM-RL, this intervention is highly sensitive to its coefficient, can introduce semantically weak uncertainty, and often yields limited accuracy gains. This motivates a more precise question: which entropy helps reasoning, and which entropy should be reduced? To study this, we parameterize the advantage estimator in Group Relative Policy Optimization (GRPO) into positive and negative outcome-conditioned channels and analyze their entropy dynamics. Our results show that positive-channel modulation raises \textit{productive entropy} associated with successful reasoning trajectories, while negative-channel modulation removes \textit{noisy entropy} associated with failed rollouts and reduces interference with correct paths. Guided by this channel-wise view, we propose \textbf{AsymGRPO}, which decouples the modulation strengths of positive and negative advantages. This enables flexible control over how the model updates across prompt difficulty levels, allowing stronger reinforcement of rare successes on harder prompts or stronger suppression of residual failures on easier prompts without forcing the two channels to share the same modulation strength. Experiments on five mathematical reasoning benchmarks show that AsymGRPO outperforms strong RLVR baselines, with consistent gains across model backbones.