Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients

TL;DR

This paper explains why normalization improves Group Relative Policy Optimization (GRPO) in reinforcement learning by analyzing local curvature and adaptive gradients, supported by theoretical and empirical results on benchmark datasets.

Contribution

It provides a theoretical analysis of normalization in GRPO from a local curvature perspective and empirically identifies training phases influenced by reward variance and feature orthogonality.

Findings

Normalization acts as an adaptive gradient in GRPO.

Theoretical convergence rate improves with normalization under mild conditions.

Empirical analysis reveals three training phases influenced by reward variance and feature orthogonality.

Abstract

Reinforcement learning (RL) has become a key driver of language model reasoning. Among RL algorithms, Group Relative Policy Optimization (GRPO) is the de facto standard, avoiding the need for a critic by using per-prompt baselines and variance normalization. Yet why and when this normalization helps remains unclear. In this work, we provide an explanation through the lens of local curvature of the sequence-level policy gradient: standard deviation normalization implements an adaptive gradient. Theoretically, under mild conditions, GRPO enjoys a strictly improved convergence rate over unnormalized REINFORCE, with gains characterized by the average within-prompt reward standard deviation across prompts and iterations. Empirically, our analysis on GSM8K and MATH benchmarks reveals three distinct training phases governed by the interplay between feature orthogonality and reward variance: (I) an early acceleration phase where high variance and orthogonality favor adaptive scaling; (II) a relatively stable transition phase; and (III) a late-stage regime where the loss of orthogonality limits further gains. Together, these results provide a principled account of when std normalization helps in GRPO, and offer broader insights into the design of critic-free RL algorithms.