Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratch

TL;DR

This paper introduces ScaleQuest, a scalable and cost-effective method for generating large mathematical reasoning datasets using lightweight models, significantly improving LLM reasoning capabilities without relying on proprietary data.

Contribution

We propose a novel two-stage question synthesis method, ScaleQuest, enabling large-scale dataset creation from scratch with lightweight models, enhancing open-source LLM reasoning performance.

Findings

Models trained on our dataset outperform existing open-source datasets in reasoning tasks.

Performance improves with increasing data volume, demonstrating scalability.

Significant gains in code reasoning tasks show strong generalization.

Abstract

Improving the mathematical reasoning capabilities of Large Language Models (LLMs) is critical for advancing artificial intelligence. However, access to extensive, diverse, and high-quality reasoning datasets remains a significant challenge, particularly for the open-source community. In this paper, we propose ScaleQuest, a novel, scalable, and cost-effective data synthesis method that enables the generation of large-scale mathematical reasoning datasets using lightweight 7B-scale models. ScaleQuest introduces a two-stage question-tuning process comprising Question Fine-Tuning (QFT) and Question Preference Optimization (QPO) to unlock the question generation capabilities of problem-solving models. By generating diverse questions from scratch -- without relying on powerful proprietary models or seed data -- we produce a dataset of 1 million problem-solution pairs. Our experiments demonstrate that models trained on our data outperform existing open-source datasets in both in-domain and out-of-domain evaluations. Furthermore, our approach shows continued performance improvement as the volume of training data increases, highlighting its potential for ongoing data scaling. The extensive improvements observed in code reasoning tasks demonstrate the generalization capabilities of our proposed method. Our work provides the open-source community with a practical solution to enhance the mathematical reasoning abilities of LLMs.