Examining False Positives under Inference Scaling for Mathematical Reasoning

TL;DR

This paper investigates the prevalence and impact of false positive solutions in mathematical reasoning by language models, revealing persistent issues across models and datasets that affect evaluation metrics and scaling behavior.

Contribution

It systematically analyzes false positives in mathematical problem solving, highlighting their persistence and influence on inference scaling and evaluation metrics.

Findings

False positives are common across models and datasets.

Sampling-based inference does not reduce false positives.

pass@N metric is more affected by false positives, indicating a lower scaling ceiling.

Abstract

Recent advancements in language models have led to significant improvements in mathematical reasoning across various benchmarks. However, most of these benchmarks rely on automatic evaluation methods that only compare final answers using heuristics, without verifying the underlying reasoning steps. This limitation results in false positive solutions, where models may produce correct final answers but with flawed deduction paths. In this paper, we systematically examine the prevalence of false positive solutions in mathematical problem solving for language models. We analyze the characteristics and extent of this issue across different open-source models, datasets of varying difficulty levels, and decoding strategies. Specifically, we explore how false positives influence the inference time scaling behavior of language models. Our experimental results reveal that: (1) false positive solutions persist across different models, datasets, and decoding methods, (2) sampling-based inference time scaling methods do not alleviate the problem, and (3) the pass@N evaluation metric is more susceptible to false positives, suggesting a significantly lower scaling ceiling than what automatic evaluations indicate. Additionally, we analyze specific instances of false positives and discuss potential limitations in self-improvement techniques and synthetic data generation under such conditions. Our data and code are publicly available at https://github.com/Wloner0809/False-Positives-in-Math.