Enhancing Reasoning through Process Supervision with Monte Carlo Tree Search

TL;DR

This paper introduces a method that uses Monte Carlo Tree Search to generate process supervision data for training large language models, significantly enhancing their reasoning capabilities and transferability across mathematical reasoning tasks.

Contribution

The paper proposes a novel approach combining MCTS with LLMs to generate process supervision data, improving reasoning performance and transferability.

Findings

Significant improvement in reasoning accuracy on two datasets

Enhanced transferability of reasoning skills across tasks

Iterative generate-then-train process converges effectively

Abstract

Large language models (LLMs) have demonstrated their remarkable capacity across a variety of tasks. However, reasoning remains a challenge for LLMs. To improve LLMs' reasoning ability, process supervision has proven to be better than outcome supervision. In this work, we study using Monte Carlo Tree Search (MCTS) to generate process supervision data with LLMs themselves for training them. We sample reasoning steps with an LLM and assign each step a score that captures its "relative correctness," and the LLM is then trained by minimizing weighted log-likelihood of generating the reasoning steps. This generate-then-train process is repeated iteratively until convergence.Our experimental results demonstrate that the proposed methods considerably improve the performance of LLMs on two mathematical reasoning datasets. Furthermore, models trained on one dataset also exhibit improved performance on the other, showing the transferability of the enhanced reasoning ability.