EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework

TL;DR

EFRame is a reinforcement learning framework that enhances reasoning in large language models by combining exploration, filtering, and replay techniques to improve training stability and performance on complex tasks.

Contribution

It introduces a novel Exploration-Filter-Replay framework that significantly improves the efficiency and stability of policy optimization for reasoning tasks in LLMs.

Findings

Achieves a 37.9% relative improvement on Geometry3K benchmark.

Supports fine-grained sample categorization and entropy control.

Demonstrates consistent gains across diverse reasoning benchmarks.

Abstract

Recent advances in reinforcement learning (RL) have significantly enhanced the reasoning capabilities of large language models (LLMs). Group Relative Policy Optimization (GRPO), a lightweight variant of Proximal Policy Optimization (PPO), improves efficiency but suffers from limited exploration and training instability, limiting its effectiveness on complex reasoning tasks. To address these challenges, we introduce EFRame, an Exploration-Filter-Replay framework that augments GRPO across three dimensions: additional rollouts enable deeper and more targeted exploration, online filtering removes low-quality samples to stabilize gradients and accelerate training, and experience replay amplifies rare yet informative trajectories for stable convergence. This unified framework establishes a principled training cycle that balances exploration, efficiency, and stability. Experiments on diverse reasoning benchmarks demonstrate that EFRame achieves consistent gains, including a 37.9\% relative improvement on Geometry3K over GRPO. EFRame further supports fine-grained sample categorization and precise entropy control, highlighting it as a robust solution for advancing deeper reasoning in LLMs. Our code is available at https://github.com/597358816/EFRame.