Group Sequence Policy Optimization

TL;DR

This paper presents GSPO, a reinforcement learning algorithm for large language models that improves training stability and efficiency by using sequence-level importance ratios, leading to better performance and simpler infrastructure.

Contribution

GSPO introduces sequence-level importance ratios and clipping for RL training of large language models, outperforming previous methods like GRPO and stabilizing MoE RL training.

Findings

GSPO achieves superior training efficiency and performance.

GSPO stabilizes Mixture-of-Experts RL training.

GSPO contributes to improvements in Qwen3 models.

Abstract

This paper introduces Group Sequence Policy Optimization (GSPO), our stable, efficient, and performant reinforcement learning algorithm for training large language models. Unlike previous algorithms that adopt token-level importance ratios, GSPO defines the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization. We demonstrate that GSPO achieves superior training efficiency and performance compared to the GRPO algorithm, notably stabilizes Mixture-of-Experts (MoE) RL training, and has the potential for simplifying the design of RL infrastructure. These merits of GSPO have contributed to the remarkable improvements in the latest Qwen3 models.