Orchestrating Tokens and Sequences: Dynamic Hybrid Policy Optimization for RLVR

TL;DR

This paper introduces Dynamic Hybrid Policy Optimization (DHPO), a novel reinforcement learning method that combines token-level and sequence-level importance ratios to improve reasoning tasks in large language models.

Contribution

DHPO effectively integrates GRPO and GSPO using a weighted surrogate objective and branch-specific clipping, enhancing training stability and performance in reasoning benchmarks.

Findings

DHPO outperforms GRPO and GSPO on seven reasoning benchmarks.

DHPO improves training stability through branch-specific clipping.

Experiments demonstrate consistent performance gains across models.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising framework for optimizing large language models in reasoning tasks. However, existing RLVR algorithms focus on different granularities, and each has complementary strengths and limitations. Group Relative Policy Optimization (GRPO) updates the policy with token-level importance ratios, which preserves fine-grained credit assignment but often suffers from high variance and instability. In contrast, Group Sequence Policy Optimization (GSPO) applies single sequence-level importance ratios across all tokens in a response that better matches sequence-level rewards, but sacrifices token-wise credit assignment. In this paper, we propose Dynamic Hybrid Policy Optimization (DHPO) to bridge GRPO and GSPO within a single clipped surrogate objective. DHPO combines token-level and sequence-level importance ratios using weighting mechanisms. We explore two variants of the mixing mechanism, including an averaged mixing and an entropy-guided mixing. To further stabilize training, we employ a branch-specific clipping strategy that constrains token-level and sequence-level ratios within separate trust regions before mixing, preventing outliers in either branch from dominating the update. Across seven challenging mathematical reasoning benchmarks, experiments on both dense and MoE models from the Qwen3 series show that DHPO consistently outperforms GRPO and GSPO. We will release our code upon acceptance of this paper.