Learning to Reason with Curriculum I: Provable Benefits of Autocurriculum

TL;DR

This paper demonstrates that autocurriculum, an adaptive training method where models select training problems based on performance, significantly reduces data and compute costs in training reasoning models, with provable benefits over standard methods.

Contribution

It introduces autocurriculum as a provably effective approach that improves training efficiency for reasoning models in both supervised and reinforcement learning settings.

Findings

Autocurriculum requires exponentially fewer demonstrations in supervised fine-tuning.

In reinforcement learning, autocurriculum decouples training cost from reference model quality.

Adaptive data selection based on model performance enhances training efficiency.

Abstract

Chain-of-thought reasoning, where language models expend additional computation by producing thinking tokens prior to final responses, has driven significant advances in model capabilities. However, training these reasoning models is extremely costly in terms of both data and compute, as it involves collecting long traces of reasoning behavior from humans or synthetic generators and further post-training the model via reinforcement learning. Are these costs fundamental, or can they be reduced through better algorithmic design? We show that autocurriculum, where the model uses its own performance to decide which problems to focus training on, provably improves upon standard training recipes for both supervised fine-tuning (SFT) and reinforcement learning (RL). For SFT, we show that autocurriculum requires exponentially fewer reasoning demonstrations than non-adaptive fine-tuning, by focusing teacher supervision on prompts where the current model struggles. For RL fine-tuning, autocurriculum decouples the computational cost from the quality of the reference model, reducing the latter to a burn-in cost that is nearly independent of the target accuracy. These improvements arise purely from adaptive data selection, drawing on classical techniques from boosting and learning from counterexamples, and requiring no assumption on the distribution or difficulty of prompts.