NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks
Yang Li, Youssef Emad, Karthik Padthe, Jack Lanchantin, Weizhe Yuan, Thao Nguyen, Jason Weston, Shang-Wen Li, Dong Wang, Ilia Kulikov, Xian Li

TL;DR
This paper introduces NaturalThoughts, a curated set of high-quality reasoning traces from a strong teacher model, which improves the reasoning capabilities of smaller models more efficiently than existing datasets.
Contribution
It systematically analyzes factors affecting reasoning trace distillation and demonstrates that selecting diverse, difficult examples enhances transfer learning for reasoning tasks.
Findings
Scaling data size with random sampling improves performance.
Selecting diverse, difficult examples increases sample efficiency.
Training with NaturalThoughts outperforms existing datasets on reasoning benchmarks.
Abstract
Recent work has shown that distilling reasoning traces from a larger teacher model via supervised finetuning outperforms reinforcement learning with the smaller student model alone (Guo et al. 2025). However, there has not been a systematic study of what kind of reasoning demonstrations from the teacher are most effective in improving the student model's reasoning capabilities. In this work we curate high-quality "NaturalThoughts" by selecting reasoning traces from a strong teacher model based on a large pool of questions from NaturalReasoning (Yuan et al. 2025). We first conduct a systematic analysis of factors that affect distilling reasoning capabilities, in terms of sample efficiency and scalability for general reasoning tasks. We observe that simply scaling up data size with random sampling is a strong baseline with steady performance gains. Further, we find that selecting…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsIntelligent Tutoring Systems and Adaptive Learning · Topic Modeling · Multimodal Machine Learning Applications
