expo: Exploration-prioritized policy optimization via adaptive kl regulation and gaussian curriculum sampling

TL;DR

This paper introduces EXPO, a reinforcement learning method that enhances policy exploration and training efficiency through adaptive KL regulation and Gaussian curriculum sampling, leading to significant performance improvements in mathematical reasoning tasks.

Contribution

The paper proposes two novel modules, AKL and GCS, to improve policy optimization by dynamically adjusting regularization and focusing on informative training samples, outperforming existing methods.

Findings

EXPO achieves a 13.34 point gain on AIME 2025 pass@32, from 63.33% to 76.67%.

EXPO improves average pass@32 by 2.66 points on 8B models.

Larger gains on pass@32 indicate better exploration boundary enlargement.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become the standard paradigm for LLM mathematical reasoning, where Group Relative Policy Optimization (GRPO) serves as the mainstream algorithm. We point out two understudied inefficiencies existing in GRPO. First, the fixed KL penalty coefficient overly restricts policy exploration at stages where the model requires significant deviation from the reference policy. Second, uniform sampling of training questions ignores that moderately difficult problems provide the most informative gradient signals for optimization. We propose Exploration-Prioritized Policy Optimization (EXPO) with two lightweight plug-in modules. The Accuracy-Conditioned KL Scaling (AKL) dynamically adjusts KL regularization strength through a smooth nonlinear function of batch average accuracy, relaxing the penalty when the model underperforms and strengthening it when the model achieves good results. The Gaussian Curriculum Sampling (GCS) assigns sampling weights to questions following a Gaussian distribution centered at moderate accuracy around 0.5, focusing training on the model's learning frontier. We conduct extensive experiments on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base over six mathematical reasoning benchmarks. The results show EXPO steadily surpasses vanilla GRPO. It obtains an absolute gain of 13.34 on AIME 2025 pass@32, rising from 63.33 percent to 76.67 percent, and achieves an average pass@32 improvement of 2.66 on the 8B model. The much larger performance gains on pass@32 compared with pass@1 demonstrate that EXPO effectively enlarges the model's exploration boundary under a fixed inference cost budget.