EBPO: Empirical Bayes Shrinkage for Stabilizing Group-Relative Policy Optimization

TL;DR

EBPO introduces a Bayesian shrinkage approach to stabilize group-relative policy optimization in reinforcement learning, reducing variance and improving training stability, especially with small groups and in failure regimes.

Contribution

The paper proposes EBPO, a novel empirical Bayes framework that enhances policy optimization stability by leveraging global statistics for baseline estimation.

Findings

EBPO reduces estimator variance compared to GRPO.

EBPO achieves higher performance across benchmarks like AIME and OlympiadBench.

EBPO maintains stability with small group sizes and in failure scenarios.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for enhancing the reasoning capabilities of Large Language Models (LLMs). However, dominant approaches like Group Relative Policy Optimization (GRPO) face critical stability challenges: they suffer from high estimator variance under computational constraints (small group sizes) and vanishing gradient signals in saturated failure regimes where all responses yield identical zero rewards. To address this, we propose Empirical Bayes Policy Optimization (EBPO), a novel framework that regularizes local group-based baselines by borrowing strength from the policy's accumulated global statistics. Instead of estimating baselines in isolation, EBPO employs a shrinkage estimator that dynamically balances local group statistics with a global prior updated via Welford's online algorithm. Theoretically, we demonstrate that EBPO guarantees strictly lower Mean Squared Error (MSE), bounded entropy decay, and non-vanishing penalty signals in failure scenarios compared to GRPO. Empirically, EBPO consistently outperforms GRPO and other established baselines across diverse benchmarks, including AIME and OlympiadBench. Notably, EBPO exhibits superior training stability, achieving high-performance gains even with small group sizes, and benefits significantly from difficulty-stratified curriculum learning.