Improving Value-based Process Verifier via Low-Cost Variance Reduction

TL;DR

This paper introduces ComMCS, a variance reduction technique for value-based process verifiers in LLM reasoning tasks, improving estimation accuracy without extra inference costs.

Contribution

The paper proposes ComMCS, a novel unbiased variance reduction method for Monte Carlo estimators, enhancing reasoning accuracy in LLM-based process verification.

Findings

ComMCS reduces variance predictably without additional inference cost.

It outperforms baseline methods by 2.2 to 2.8 points on MATH-500.

Empirical results validate the theoretical variance reduction benefits.

Abstract

Large language models (LLMs) have achieved remarkable success in a wide range of tasks. However, their reasoning capabilities, particularly in complex domains like mathematics, remain a significant challenge. Value-based process verifiers, which estimate the probability of a partial reasoning chain leading to a correct solution, are a promising approach for improving reasoning. Nevertheless, their effectiveness is often hindered by estimation error in their training annotations, a consequence of the limited number of Monte Carlo (MC) samples feasible due to the high cost of LLM inference. In this paper, we identify that the estimation error primarily arises from high variance rather than bias, and the MC estimator is a Minimum Variance Unbiased Estimator (MVUE). To address the problem, we propose the \textsc{Com}pound \textsc{M}onte \textsc{C}arlo \textsc{S}ampling (ComMCS) method, which constructs an unbiased estimator by linearly combining the MC estimators from the current and subsequent steps. Theoretically, we show that our method leads to a predictable reduction in variance, while maintaining an unbiased estimation without additional LLM inference cost. We also perform empirical experiments on the MATH-500 and GSM8K benchmarks to demonstrate the effectiveness of our method. Notably, ComMCS outperforms regression-based optimization method by 2.8 points, the non-variance-reduced baseline by 2.2 points on MATH-500 on Best-of-32 sampling experiment.