CMCTS: A Constrained Monte Carlo Tree Search Framework for Mathematical Reasoning in Large Language Model

TL;DR

The paper presents CMCTS, a novel constrained Monte Carlo Tree Search framework that significantly improves mathematical reasoning in large language models by incorporating action constraints, process reward modeling, and partial order rules, leading to higher accuracy.

Contribution

Introduces CMCTS, a new MCTS framework with constraints, PRM, and partial order rules to enhance LLM reasoning, outperforming baseline models on benchmarks.

Findings

Achieves 83.4% accuracy with a 7B model in zero-shot settings.

Outperforms larger baseline models by 4.8%.

Each component of CMCTS is crucial for performance gains.

Abstract

This paper introduces the Constrained Monte Carlo Tree Search (CMCTS) framework to enhance the mathematical reasoning capabilities of Large Language Models (LLM). By incorporating a constrained action space, Process Reward Model (PRM), and partial order rules, CMCTS effectively addresses the limitations of existing MCTS methods in terms of state space diversity and action selection rationality. Specifically, during the expansion phase, CMCTS restricts action sampling to a predefined constrained action set to increase candidate state diversity. In the simulation phase, it introduces partial order rules and PRM to optimize action selection and prevent unreasonable state transitions. Experimental results show that CMCTS performs outstandingly across multiple mathematical reasoning benchmarks. Under a zero-shot setting, a 7B-parameter model achieves an average accuracy of 83.4\%, surpassing the 72B baseline model by 4.8\%. Ablation studies demonstrate that each component of the framework is crucial for performance improvement, and their combined use fully leverages their respective strengths. Overall, the CMCTS framework provides an effective approach to enhancing LLM mathematical reasoning capabilities, supported by theoretical analysis, and offers novel insights for future reasoning tasks.