FastMCTS: A Simple Sampling Strategy for Data Synthesis

TL;DR

FastMCTS is an efficient Monte Carlo Tree Search-inspired method for generating high-quality multi-step reasoning data, outperforming rejection sampling in accuracy and diversity, thereby enhancing large language model performance.

Contribution

Introduces FastMCTS, a novel, efficient sampling strategy for reasoning data synthesis that improves over rejection sampling by providing balanced, step-level evaluation signals.

Findings

Generates 30% more correct reasoning paths than rejection sampling.

Models trained on FastMCTS data outperform those trained on rejection sampling data by 3.9%.

FastMCTS is a lightweight, practical alternative for high-quality reasoning data synthesis.

Abstract

Synthetic high-quality multi-step reasoning data can significantly enhance the performance of large language models on various tasks. However, most existing methods rely on rejection sampling, which generates trajectories independently and suffers from inefficiency and imbalanced sampling across problems of varying difficulty. In this work, we introduce FastMCTS, an innovative data synthesis strategy inspired by Monte Carlo Tree Search. FastMCTS provides a more efficient sampling method for multi-step reasoning data, offering step-level evaluation signals and promoting balanced sampling across problems of different difficulty levels. Experiments on both English and Chinese reasoning datasets demonstrate that FastMCTS generates over 30\% more correct reasoning paths compared to rejection sampling as the number of generated tokens scales up. Furthermore, under comparable synthetic data budgets, models trained on FastMCTS-generated data outperform those trained on rejection sampling data by 3.9\% across multiple benchmarks. As a lightweight sampling strategy, FastMCTS offers a practical and efficient alternative for synthesizing high-quality reasoning data. Our code will be released soon.