The Challenge of Teaching Reasoning to LLMs Without RL or Distillation

TL;DR

This paper investigates inducing reasoning capabilities in base language models through minimal prompting and fine-tuning with high-quality Chain-of-Thought examples, showing promising results with small datasets.

Contribution

It demonstrates that a small number of carefully curated CoT examples can significantly enhance reasoning in base models without reinforcement learning or large-scale distillation.

Findings

Light fine-tuning with few high-quality CoT examples improves reasoning.

CoT data from non-reasoning models or humans is less effective than expert traces.

Curated reasoning data influences problem difficulty, diversity, and answer length.

Abstract

Reasoning-capable language models achieve state-of-the-art performance in diverse complex tasks by generating long, explicit Chain-of-Thought (CoT) traces. While recent works show that base models can acquire such reasoning traces via reinforcement learning or distillation from stronger models like DeepSeek-R1, previous works demonstrate that even short CoT prompting without fine-tuning is able to improve reasoning. We ask whether long CoT can be induced in a base model using only prompting or minimal tuning. Using just 20 long CoT examples from the reasoning model \texttt{QwQ-32B-Preview}, we lightly fine-tune the base model \texttt{Qwen2.5-32B}. The resulting model outperforms the much larger \texttt{Qwen2.5-Math-72B-Instruct}, showing that a handful of high-quality examples can unlock strong reasoning capabilities. We further explore using CoT data from non-reasoning models and human annotators, enhanced with prompt engineering, multi-pass editing, and structural guidance. However, neither matches the performance of reasoning model traces, suggesting that certain latent qualities of expert CoT are difficult to replicate. We analyze key properties of reasoning data, such as problem difficulty, diversity, and answer length, that influence reasoning distillation. While challenges remain, we are optimistic that carefully curated human-written CoT, even in small quantities, can activate reasoning behaviors in base models. We release our human-authored dataset across refinement stages and invite further investigation into what makes small-scale reasoning supervision so effective.