Let's Verify Math Questions Step by Step

TL;DR

This paper introduces ValiMath, a high-quality benchmark of 2147 verified math questions, and MathQ-Verify, a pipeline for parsing and verifying question correctness to improve dataset quality for LLM training.

Contribution

The paper presents ValiMath as a new benchmark for question correctness and introduces MathQ-Verify, a novel method for fine-grained parsing and semantic verification of math questions.

Findings

ValiMath provides a reliable gold-standard dataset for math question evaluation.

MathQ-Verify achieves state-of-the-art accuracy in question verification tasks.

The pipeline significantly reduces noise in mathematical datasets, enhancing LLM training quality.

Abstract

Large Language Models (LLMs) have recently achieved remarkable progress in mathematical reasoning. To enable such capabilities, many existing works distill strong reasoning models into long chains of thought or design algorithms to construct high-quality math question-answer (QA) data for training. However, these efforts primarily focus on generating correct reasoning paths and answers, while largely overlooking the correctness of the questions themselves. In this work, we present ValiMath, a benchmark consisting of 2147 human-verified mathematical questions covering a wide range of domains such as arithmetic, algebra, and geometry, which are synthesized and curated from the NuminaMath dataset. Each question is annotated with its logical structure, domain coverage, and question correctness, enabling fine-grained evaluation of question quality. ValiMath serves as a high-quality gold-standard test set for validating mathematical questions in LLM training corpora. Building upon this benchmark, we further propose MathQ-Verify, a pipeline that performs fine-grained parsing of mathematical questions into atomic assumptions and conclusions, and evaluates their semantic soundness through consistency checks. This pipeline achieves high precision in detecting flawed questions and provides a reliable foundation for cleaning noisy mathematical datasets. Experiments show that MathQ-Verify achieves state-of-the-art performance across multiple benchmarks, improving the F1 score by up to 25 percentage points over the direct verification baseline. MathQ-Verify offers a scalable and accurate solution for curating reliable mathematical datasets, reducing label noise and avoiding unnecessary computation on invalid questions. Our code and data are available at the repository https://github.com/OpenDCAI/MathQ-Verify.