Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning

TL;DR

This paper introduces Group Relative Reward Rescaling (GR³), a novel method to control length inflation in reinforcement learning for language models, maintaining performance while reducing verbosity.

Contribution

The paper proposes GR³, a generalized, reward-dependent gating mechanism with regularization and calibration to effectively mitigate length inflation without loss of training quality.

Findings

GR³ effectively reduces length inflation in RLHF and RLVR settings.

Maintains comparable training dynamics and downstream performance to standard methods.

Outperforms existing length-regularized baselines.

Abstract

Reinforcement learning significantly enhances LLM capabilities but suffers from a critical issue: length inflation, where models adopt verbosity or inefficient reasoning to maximize rewards. Prior approaches struggle to address this challenge in a general and lossless manner, primarily because additive penalties introduce a compensatory effect that creates optimization shortcuts, while heuristic gating strategies lack generality beyond binary feedback. To bridge this gap, we present Group Relative Reward Rescaling (GR$^3$), which reframes length control as a multiplicative rescaling paradigm, effectively establishing a generalized, continuous, and reward-dependent gating mechanism. To further ensure lossless optimization, we incorporate group-relative regularization and advantage-aware calibration, which dynamically adapt length budgets to instance difficulty and preserve the advantage signal of high-quality trajectories. Empirically, across both RLHF and RLVR settings, GR$^3$~maintains training dynamics and downstream performance comparable to standard GRPO while significantly mitigating length inflation, outperforming state-of-the-art length-regularized baselines.