ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models

TL;DR

This paper introduces ReliableMath, a benchmark for assessing the reliability of large language models in mathematical reasoning, especially on solvable and unsolvable problems, and proposes strategies to improve their reliability.

Contribution

The paper develops a new dataset and evaluation framework for LLM reliability in math reasoning, and proposes an alignment strategy to enhance small LLMs' reliability.

Findings

LLMs often fabricate responses and fail to identify unsolvable problems.

Larger LLMs show improved reliability with reliable prompts, especially on unsolvable problems.

Small LLMs benefit significantly from the proposed alignment strategy.

Abstract

Although demonstrating remarkable performance on reasoning tasks, Large Language Models (LLMs) still tend to fabricate unreliable responses when confronted with problems that are unsolvable or beyond their capability, severely undermining the reliability. Prior studies of LLM reliability have primarily focused on knowledge tasks to identify unanswerable questions, while mathematical reasoning tasks have remained unexplored due to the dearth of unsolvable math problems. To systematically investigate LLM reliability in mathematical reasoning tasks, we formulate the reliability evaluation for both solvable and unsolvable problems. We then develop a ReliableMath dataset which incorporates open-source solvable problems and high-quality unsolvable problems synthesized by our proposed construction workflow with human evaluations. Experiments are conducted on various LLMs with several key findings uncovered. LLMs fail to directly identify unsolvable problems and always generate fabricated responses. When instructing LLMs to indicate unsolvability using a reliable prompt, the reliability of larger-sized LLMs remains on solvable problems, but notably improves on unsolvable problems yet still falls short of solvable problems. However, small LLMs rarely show any progress despite employing reliable prompts. Therefore, we further propose an alignment strategy to enhance small LLMs' reliability, which can significantly improve LLM reliability performances on both in-domain and out-of-domain tasks.