MathClean: A Benchmark for Synthetic Mathematical Data Cleaning

TL;DR

MathClean is a new benchmark designed to evaluate and improve the effectiveness of models in cleaning synthetic mathematical data, which is crucial for training large language models with strong reasoning abilities.

Contribution

The paper introduces MathClean, a comprehensive benchmark with annotated error types for synthetic math data, and evaluates state-of-the-art models, revealing their current limitations.

Findings

State-of-the-art models perform poorly on MathClean

MathClean includes 8,000 questions with error annotations

Benchmark highlights need for improved data cleaning methods

Abstract

With the rapid development of large language models (LLMs), the quality of training data has become crucial. Among the various types of training data, mathematical data plays a key role in enabling LLMs to acquire strong reasoning abilities. While high-quality open-source data is important, it is often insufficient for pre-training, necessitating the addition of synthetic math problems. However, synthetic math questions and answers can introduce inaccuracies, which may degrade both the training data and web data. Therefore, an effective method for cleaning synthetic math data is essential. In this paper, we propose the MathClean benchmark to evaluate the effectiveness of math data cleaning models. The MathClean benchmark consists of 2,000 correct questions and 2,000 erroneous questions with additional 2,000 correct and erroneous answers sourced from augmented data based on GSM8K and MATH. Moreover, we also annotate error types for each question or answer, since it can assess whether models can correctly identify the error categories for future improvements. Finally, we present comprehensive evaluations using state-of-the-art (SOTA) models. Our results demonstrate that even strong models like GPT-o1 and DeepSeek-R1 perform poorly on this benchmark, highlighting the utility of MathClean. Our code and data is available at https://github.com/YuYingLi0/MathClean.