Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works

TL;DR

This paper identifies gradient starvation as a failure mode in group-mean-centered advantage methods for binary-reward reinforcement learning and proposes a simple fix that significantly improves performance.

Contribution

The authors analyze the failure mode of group-mean advantage in binary rewards and introduce a fixed-reference Sign advantage that mitigates gradient starvation.

Findings

Sign advantage achieves 73.8% accuracy on GSM8K test set, outperforming standard methods.

Gradient starvation occurs with a degeneracy rate of 0.69 at group size four.

The proposed fix improves search efficiency rather than capacity.

Abstract

Group Relative Policy Optimization (GRPO) is a standard algorithm for reinforcement learning from verifiable rewards, but its group-mean-centered advantage can fail under binary rewards. The failure mode is gradient starvation: when every response in a group is correct or every response is wrong, the centered advantage is exactly zero and the policy receives no learning signal. We prove that the true degeneracy rate always exceeds the i.i.d. Bernoulli prediction by Jensen's inequality, and observe a 0.69 degeneracy rate at group size four in logged Qwen3.5-9B GSM8K training. We then show that the fixed-reference Sign advantage, $A=2r-1$, performs pass@$G$ failure descent by increasing the probability that at least one sample in the group succeeds. On the full GSM8K test set across seven seeds, Sign reaches 73.8% accuracy versus 28.4% for standard normalized group-mean DrGRPO at group size four, a 45.4 point gain with $p<0.0001$. The effect is directionally consistent on Llama-3.1-8B and positive but underpowered on a MATH-500 transfer check. Pass@$k$ analysis indicates that the main benefit is search compression rather than large capacity expansion, aligning the empirical gains with recent RLVR ceiling observations.