Evaluating Mathematical Reasoning Beyond Accuracy

TL;DR

This paper introduces ReasonEval, a new methodology for evaluating the quality of reasoning steps in large language models' mathematical tasks, going beyond just final answer accuracy.

Contribution

The paper presents ReasonEval, an automatic evaluation framework for reasoning quality that outperforms baselines and demonstrates strong generalization in mathematical reasoning tasks.

Findings

ReasonEval outperforms baseline methods in meta-evaluation datasets.

Higher final-answer accuracy does not always mean better reasoning quality.

ReasonEval can effectively guide data selection and improve reasoning assessment.

Abstract

The leaderboard of Large Language Models (LLMs) in mathematical tasks has been continuously updated. However, the majority of evaluations focus solely on the final results, neglecting the quality of the intermediate steps. This oversight can mask underlying problems, such as logical errors or unnecessary steps in the reasoning process. To measure reasoning beyond final-answer accuracy, we introduce ReasonEval, a new methodology for evaluating the quality of reasoning steps. ReasonEval employs validity and redundancy to characterize the reasoning quality, as well as accompanying LLMs to assess them automatically. We explore different design options for the LLM-based evaluators and empirically demonstrate that ReasonEval, when instantiated with base models possessing strong mathematical knowledge and trained with high-quality labeled data, consistently outperforms baseline methods in the meta-evaluation datasets. We also highlight the strong generalization capabilities of ReasonEval. By utilizing ReasonEval to evaluate LLMs specialized in math, we find that an increase in final-answer accuracy does not necessarily guarantee an improvement in the overall quality of the reasoning steps for challenging mathematical problems. Additionally, we observe that ReasonEval can play a significant role in data selection. We open-source the best-performing model, meta-evaluation script, and all evaluation results to facilitate future research.