AlphaMath Almost Zero: Process Supervision without Process

TL;DR

AlphaMath introduces a process supervision method for mathematical reasoning in LLMs that uses Monte Carlo Tree Search and a value model, eliminating the need for costly human or GPT-4 annotations, and achieves state-of-the-art results.

Contribution

It presents a novel framework that enables LLMs to improve mathematical reasoning autonomously without human or GPT-4 process annotations, using MCTS and a value model.

Findings

Achieves comparable or superior results to state-of-the-art methods.

Operates effectively without human or GPT-4 process supervision.

Demonstrates strong performance on in-domain and out-of-domain datasets.

Abstract

Although recent advancements in large language models (LLMs) have significantly improved their performance on various tasks, they still face challenges with complex and symbolic multi-step reasoning, particularly in mathematical reasoning. To bolster the mathematical reasoning capabilities of LLMs, most existing efforts concentrate on seeking assistance from either domain experts or GPT-4 for high-quality process-supervised data, which is not only expensive but also labor-intensive. In our study, we propose an innovative framework, AlphaMath, that bypasses the need for process annotations (from humans or GPTs) by leveraging Monte Carlo Tree Search (MCTS). This framework focuses on unleashing the potential of a well-pretrained LLM to autonomously enhance its mathematical reasoning. Specifically, we integrate a value model with the LLM, automatically generating both process supervision and step-level evaluation signals in MCTS. Furthermore, we propose an efficient inference strategy, step-level beam search, where the value model is crafted to assist the policy model (i.e., LLM) in navigating more effective reasoning paths, rather than solely relying on prior probabilities. The experimental results on both in-domain and out-of-domain datasets demonstrate that even without GPT-4 or human-annotated process supervision, our AlphaMath framework achieves comparable or superior results to previous state-of-the-art methods.