Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlo

TL;DR

This paper introduces Twisted Sequential Monte Carlo (TSMC), a novel verification method that enhances multi-step reasoning in Large Language Models for math problems by improving sampling efficiency and reducing supervision needs.

Contribution

The paper proposes TSMC, a new verification approach that refines sampling on promising candidates, enabling more efficient solution generation without extensive step-wise annotations.

Findings

TSMC outperforms existing verification methods on multiple math benchmarks.

Theoretical analysis confirms the efficiency and effectiveness of TSMC.

TSMC reduces the need for costly human annotations in training.

Abstract

Augmenting the multi-step reasoning abilities of Large Language Models (LLMs) has been a persistent challenge. Recently, verification has shown promise in improving solution consistency by evaluating generated outputs. However, current verification approaches suffer from sampling inefficiencies, requiring a large number of samples to achieve satisfactory performance. Additionally, training an effective verifier often depends on extensive process supervision, which is costly to acquire. In this paper, we address these limitations by introducing a novel verification method based on Twisted Sequential Monte Carlo (TSMC). TSMC sequentially refines its sampling effort to focus exploration on promising candidates, resulting in more efficient generation of high-quality solutions. We apply TSMC to LLMs by estimating the expected future rewards at partial solutions. This approach results in a more straightforward training target that eliminates the need for step-wise human annotations. We empirically demonstrate the advantages of our method across multiple math benchmarks, and also validate our theoretical analysis of both our approach and existing verification methods.