F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

TL;DR

This paper introduces a difficulty-aware advantage scaling method for group-based reinforcement learning, reducing the bias towards common solutions and improving performance on benchmarks without extra computational cost.

Contribution

It derives the probability of missing rare trajectories in group sampling and proposes a simple, effective modification inspired by Focal loss to address this issue.

Findings

Improves pass@256 on multiple benchmarks

Reduces bias towards common solutions

Maintains or improves pass@1 performance

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is commonly based on group sampling to estimate advantages and stabilize policy updates. In practice, large group sizes are not feasible due to computational limits, which biases learning toward trajectories that are already likely. Smaller groups often miss rare-correct trajectories while still containing mixed rewards, concentrating probability on common solutions. We derive the probability that updates miss rare-correct modes as a function of group size, showing non-monotonic behavior, and characterize how updates redistribute mass within the correct set, revealing that unsampled-correct mass can shrink even as total correct mass grows. Motivated by this analysis, we propose a difficulty-aware advantage scaling coefficient, inspired by Focal loss, that down-weights updates on high-success prompts. The lightweight modification can be directly integrated into any group-relative RLVR algorithm such as GRPO, DAPO, and CISPO. On Qwen2.5-7B across in-domain and out-of-domain benchmarks, our method improves pass@256 from 64.1 $\rightarrow$ 70.3 (GRPO), 69.3 $\rightarrow$ 72.5 (DAPO), and 73.2 $\rightarrow$ 76.8 (CISPO), while preserving or improving pass@1, without increasing group size or computational cost.