CPGD: Toward Stable Rule-based Reinforcement Learning for Language Models

TL;DR

This paper introduces CPGD, a new algorithm for rule-based reinforcement learning in language models that enhances training stability and performance by regulating policy updates with KL divergence constraints and clipping mechanisms.

Contribution

The paper proposes CPGD, a novel stabilization method for RL in language models, combining KL divergence-based regularization and clipping to prevent training collapse.

Findings

CPGD reduces training instability in RL for language models.

Empirical results show improved performance over existing methods.

Theoretical analysis supports the stability benefits of CPGD.

Abstract

Recent advances in rule-based reinforcement learning (RL) have significantly improved the reasoning capability of language models (LMs) with rule-based rewards. However, existing RL methods -- such as GRPO, REINFORCE++, and RLOO -- often suffer from training instability, where large policy updates and improper clipping can lead to training collapse. To address this issue, we propose Clipped Policy Gradient Optimization with Policy Drift (CPGD), a novel algorithm designed to stabilize policy learning in LMs. CPGD introduces a policy drift constraint based on KL divergence to dynamically regularize policy updates, and leverages a clip mechanism on the logarithm of the ratio to prevent excessive policy updates. We provide theoretical justification for CPGD and demonstrate through empirical analysis that it mitigates the instability observed in prior approaches. Furthermore, we show that CPGD significantly improves performance while maintaining training stability. Our implementation balances theoretical rigor with practical usability, offering a robust alternative for RL in the post-training of LMs. We release our code at https://github.com/ModalMinds/MM-EUREKA.