VCSearch: Bridging the Gap Between Well-Defined and Ill-Defined Problems in Mathematical Reasoning

TL;DR

This paper introduces VCSearch, a training-free framework that enhances large language models' ability to identify ill-defined mathematical problems with missing or contradictory information, improving robustness and accuracy.

Contribution

The paper presents VCSearch, a novel variable-constraint search framework that effectively detects ill-defined problems without additional training, addressing limitations of existing methods.

Findings

VCSearch improves problem identification accuracy by at least 12%.

The benchmark PMC contains over 5,000 validated ill-defined problems.

Traditional methods face a trade-off between accuracy and rejection capabilities.

Abstract

Large language models (LLMs) have demonstrated impressive performance on reasoning tasks, including mathematical reasoning. However, the current evaluation mostly focuses on carefully constructed benchmarks and neglects the consideration of real-world reasoning problems that present missing or contradictory conditions, known as ill-defined problems. To further study this problem, we develop a largescale benchmark called Problems with Missing and Contradictory conditions (PMC) containing over 5,000 validated ill-defined mathematical problems. Our preliminary experiments through PMC reveal two challenges about existing methods: (1) traditional methods exhibit a trade-off between solving accuracy and rejection capabilities, and (2) formal methods struggle with modeling complex problems. To address these challenges, We develop Variable-Constraint Search (VCSEARCH), a trainingfree framework that leverages formal language to detect ill-defined problems, where a variableconstraint pair search strategy is incorporated to improve the modeling capability of formal language. Extensive experiments demonstrate that VCSEARCH improves the accuracy of identifying unsolvable problems by at least 12% across different LLMs, thus achieving stronger robust mathematical reasoning ability.