Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training

TL;DR

This paper demonstrates that prolonged reinforcement learning with specific techniques significantly enhances reasoning abilities in small language models across diverse tasks, emphasizing the importance of reward signals and training stability.

Contribution

It introduces effective training techniques like KL regularization and reference policy resets, showing substantial performance gains in reasoning tasks.

Findings

+14.7% on math tasks

+13.9% on coding tasks

+54.8% on logic puzzles

Abstract

Recent advancements in reasoning-focused language models such as OpenAI's O1 and DeepSeek-R1 have shown that scaling test-time computation-through chain-of-thought reasoning and iterative exploration-can yield substantial improvements on complex tasks like mathematics and code generation. These breakthroughs have been driven by large-scale reinforcement learning (RL), particularly when combined with verifiable reward signals that provide objective and grounded supervision. In this report, we investigate the effects of prolonged reinforcement learning on a small language model across a diverse set of reasoning domains. Our work identifies several key ingredients for effective training, including the use of verifiable reward tasks, enhancements to Group Relative Policy Optimization (GRPO), and practical techniques to improve training stability and generalization. We introduce controlled KL regularization, clipping ratio, and periodic reference policy resets as critical components for unlocking long-term performance gains. Our model achieves significant improvements over strong baselines, including +14.7% on math, +13.9% on coding, and +54.8% on logic puzzle tasks. To facilitate continued research, we release our model publicly.