One Ring to Rule Them All: Unifying Group-Based RL via Dynamic Power-Mean Geometry

TL;DR

This paper introduces Power-Mean Policy Optimization (PMPO), a unified framework for group-based reinforcement learning that adaptively adjusts the aggregation geometry to improve stability and performance across diverse trajectories.

Contribution

The paper unifies existing group-based RL methods under a flexible power-mean framework and proposes an adaptive mechanism to optimize the aggregation geometry dynamically.

Findings

PMPO outperforms strong baselines on mathematical reasoning benchmarks.

Adaptive adjustment of the aggregation geometry improves stability and learning efficiency.

Theoretical analysis shows how the parameter p influences gradient concentration and trajectory weighting.

Abstract

Group-based reinforcement learning has evolved from the arithmetic mean of GRPO to the geometric mean of GMPO. While GMPO improves stability by constraining a conservative objective, it shares a fundamental limitation with GRPO: reliance on a fixed aggregation geometry that ignores the evolving and heterogeneous nature of each trajectory. In this work, we unify these approaches under Power-Mean Policy Optimization (PMPO), a generalized framework that parameterizes the aggregation geometry via the power-mean geometry exponent p. Within this framework, GRPO and GMPO are recovered as special cases. Theoretically, we demonstrate that adjusting p modulates the concentration of gradient updates, effectively reweighting tokens based on their advantage contribution. To determine p adaptively, we introduce a Clip-aware Effective Sample Size (ESS) mechanism. Specifically, we propose a deterministic rule that maps a trajectory clipping fraction to a target ESS. Then, we solve for the specific p to align the trajectory induced ESS with this target one. This allows PMPO to dynamically transition between the aggressive arithmetic mean for reliable trajectories and the conservative geometric mean for unstable ones. Experiments on multiple mathematical reasoning benchmarks demonstrate that PMPO outperforms strong baselines.