Adaptive-Boundary-Clipping GRPO: Ensuring Bounded Ratios for Stable and Generalizable Training

TL;DR

This paper introduces ABC-GRPO, an adaptive refinement of GRPO for reinforcement learning with large language models, which improves performance, exploration, and generalization by optimizing the clipping mechanism.

Contribution

We propose ABC-GRPO, an adaptive and asymmetric modification of GRPO, enhancing its flexibility and effectiveness for training large language models.

Findings

ABC-GRPO outperforms standard GRPO on mathematical reasoning tasks.

It maintains higher entropy during training, promoting exploration.

Demonstrates improved stability and generalization in LLM training.

Abstract

Group Relative Policy Optimization (GRPO) has emerged as a popular algorithm for reinforcement learning with large language models (LLMs). However, upon analyzing its clipping mechanism, we argue that it is suboptimal in certain scenarios. With appropriate modifications, GRPO can be significantly enhanced to improve both flexibility and generalization. To this end, we propose Adaptive-Boundary-Clipping GRPO (ABC-GRPO), an asymmetric and adaptive refinement of the original GRPO framework. We demonstrate that ABC-GRPO achieves superior performance over standard GRPO on mathematical reasoning tasks using the Qwen3 LLMs. Moreover, ABC-GRPO maintains substantially higher entropy throughout training, thereby preserving the model's exploration capacity and mitigating premature convergence. The implementation code is available online to ease reproducibility https://github.com/chi2liu/ABC-GRPO.