Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning

TL;DR

This paper introduces E2H Reasoner, a curriculum learning approach that schedules tasks from easy to hard to enhance reasoning in small language models, supported by theoretical guarantees and empirical results.

Contribution

The paper proposes a novel curriculum learning method for RL in LLMs, with convergence guarantees and sample complexity analysis, improving reasoning abilities of small models.

Findings

E2H Reasoner improves reasoning in small LLMs.

Fading easy tasks prevents overfitting.

Theoretical convergence guarantees are established.

Abstract

We aim to improve the reasoning capabilities of language models via reinforcement learning (RL). Recent RL post-trained models like DeepSeek-R1 have demonstrated reasoning abilities on mathematical and coding tasks. However, prior studies suggest that using RL alone to improve reasoning on inherently difficult tasks is less effective. Here, we draw inspiration from curriculum learning and propose to schedule tasks from easy to hard (E2H), allowing LLMs to build reasoning skills gradually. Our method is termed E2H Reasoner. Empirically, we observe that, although easy tasks are important initially, fading them out through appropriate scheduling is essential in preventing overfitting. Theoretically, we establish convergence guarantees for E2H Reasoner within an approximate policy iteration framework. We derive finite-sample complexity bounds and show that when tasks are appropriately decomposed and conditioned, learning through curriculum stages requires fewer total samples than direct learning. Experiments across multiple domains show that E2H Reasoner significantly improves the reasoning ability of small LLMs (1.5B to 3B), which otherwise struggle when trained with vanilla RL alone, highlighting the effectiveness of our method. Our code can be found on https://github.com/divelab/E2H-Reasoning.