Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2

TL;DR

This paper shows that a carefully designed curriculum can significantly improve reasoning abilities and training efficiency in small language models like GPT-2, by progressing through increasingly complex reasoning tasks.

Contribution

It introduces a four-stage curriculum for training small language models that enhances reasoning, efficiency, and interpretability without task-specific fine-tuning.

Findings

Faster target accuracy with fewer training steps

More gradient-salient reasoning heads activated

Heads shift to deeper layers with higher entropy

Abstract

We demonstrate that a developmentally ordered curriculum markedly improves reasoning transparency and sample-efficiency in small language models (SLMs). Concretely, we train Cognivolve, a 124 M-parameter GPT-2 model, on a four-stage syllabus that ascends from lexical matching to multi-step symbolic inference and then evaluate it without any task-specific fine-tuning. Cognivolve reaches target accuracy in half the optimization steps of a single-phase baseline, activates an order-of-magnitude more gradient-salient reasoning heads, and shifts those heads toward deeper layers, yielding higher-entropy attention that balances local and long-range context. The same curriculum applied out of order or with optimizer resets fails to reproduce these gains, confirming that progression--not extra compute--drives the effect. We also identify open challenges: final-answer success still lags a conventional run by about 30%, and our saliency probe under-detects verbal-knowledge heads in the hardest stage, suggesting directions for mixed-stage fine-tuning and probe expansion.