Train Long, Think Short: Curriculum Learning for Efficient Reasoning

TL;DR

This paper introduces a curriculum learning approach for length-controlled reasoning in large language models, gradually tightening token budgets during training to improve accuracy and efficiency.

Contribution

It proposes a novel curriculum learning strategy using GRPO that dynamically adjusts reasoning length constraints, enhancing model performance over fixed-budget methods.

Findings

Outperforms fixed-budget baselines in accuracy and token efficiency.

Gradual tightening of reasoning length acts as an effective inductive bias.

Reward balancing improves training effectiveness.

Abstract

Recent work on enhancing the reasoning abilities of large language models (LLMs) has introduced explicit length control as a means of constraining computational cost while preserving accuracy. However, existing approaches rely on fixed-length training budgets, which do not take advantage of the natural progression from exploration to compression during learning. In this work, we propose a curriculum learning strategy for length-controlled reasoning using Group Relative Policy Optimization (GRPO). Our method starts with generous token budgets and gradually tightens them over training, encouraging models to first discover effective solution strategies and then distill them into more concise reasoning traces. We augment GRPO with a reward function that balances three signals: task correctness (via verifier feedback), length efficiency, and formatting adherence (via structural tags). Experiments on GSM8K, MATH500, SVAMP, College Math, and GSM+ demonstrate that curriculum-based training consistently outperforms fixed-budget baselines at the same final budget, achieving higher accuracy and significantly improved token efficiency. We further ablate the impact of reward weighting and decay schedule design, showing that progressive constraint serves as a powerful inductive bias for training efficient reasoning models. Our code and checkpoints are released at: https://github.com/hammoudhasan/curriculum_grpo.