SuperCLUE-Math6: Graded Multi-Step Math Reasoning Benchmark for LLMs in Chinese

TL;DR

SuperCLUE-Math6 is a new Chinese math reasoning benchmark with over 2000 problems designed to evaluate and compare the multi-step reasoning abilities of large language models, highlighting their performance stratification.

Contribution

It introduces a challenging, diverse Chinese math reasoning dataset and a novel scheme to quantify reasoning capabilities across different complexity levels.

Findings

Top models like GPT-4 outperform others.

Clear stratification of reasoning levels among models.

Fills a gap in Chinese mathematical reasoning benchmarks.

Abstract

We introduce SuperCLUE-Math6(SC-Math6), a new benchmark dataset to evaluate the mathematical reasoning abilities of Chinese language models. SC-Math6 is designed as an upgraded Chinese version of the GSM8K dataset with enhanced difficulty, diversity, and application scope. It consists of over 2000 mathematical word problems requiring multi-step reasoning and providing natural language solutions. We propose an innovative scheme to quantify the reasoning capability of large models based on performance over problems with different reasoning steps. Experiments on 13 representative Chinese models demonstrate a clear stratification of reasoning levels, with top models like GPT-4 showing superior performance. SC-Math6 fills the gap in Chinese mathematical reasoning benchmarks and provides a comprehensive testbed to advance the intelligence of Chinese language models.