R^3: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learning

TL;DR

The paper introduces R^3, a reinforcement learning framework for large reasoning models that enhances training stability and performance through replay, reflection, and ranking rewards, leading to state-of-the-art results in math benchmarks.

Contribution

It proposes a novel RL mechanism with cross-context replay, self-reflection, and entropy ranking rewards to improve large reasoning models' training and reasoning capabilities.

Findings

Achieves state-of-the-art performance on math benchmarks.

Demonstrates significant improvements over base models.

Requires fewer reasoning tokens for accurate solutions.

Abstract

Large reasoning models (LRMs) aim to solve diverse and complex problems through structured reasoning. Recent advances in group-based policy optimization methods have shown promise in enabling stable advantage estimation without reliance on process-level annotations. However, these methods rely on advantage gaps induced by high-quality samples within the same batch, which makes the training process fragile and inefficient when intra-group advantages collapse under challenging tasks. To address these problems, we propose a reinforcement learning mechanism named \emph{\textbf{R^3}} that along three directions: (1) a \emph{cross-context \underline{\textbf{R}}eplay} strategy that maintains the intra-group advantage by recalling valuable examples from historical trajectories of the same query, (2) an \emph{in-context self-\underline{\textbf{R}}eflection} mechanism enabling models to refine outputs by leveraging past failures, and (3) a \emph{structural entropy \underline{\textbf{R}}anking reward}, which assigns relative rewards to truncated or failed samples by ranking responses based on token-level entropy patterns, capturing both local exploration and global stability. We implement our method on Deepseek-R1-Distill-Qwen-1.5B and train it on the DeepscaleR-40k in the math domain. Experiments demonstrate our method achieves SoTA performance on several math benchmarks, representing significant improvements and fewer reasoning tokens over the base models. Code and model will be released.