ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection

TL;DR

ErrorRadar is a new benchmark designed to evaluate multimodal large language models' ability to detect and categorize errors in complex mathematical reasoning tasks, highlighting current performance gaps.

Contribution

The paper introduces ErrorRadar, the first benchmark for multimodal error detection in mathematical reasoning, with a comprehensive dataset and evaluation framework.

Findings

GPT-4o performs about 10% below human evaluators.

Current MLLMs face significant challenges in complex error detection tasks.

ErrorRadar reveals substantial room for improvement in multimodal mathematical reasoning.

Abstract

As the field of Multimodal Large Language Models (MLLMs) continues to evolve, their potential to revolutionize artificial intelligence is particularly promising, especially in addressing mathematical reasoning tasks. Current mathematical benchmarks predominantly focus on evaluating MLLMs' problem-solving ability, yet there is a crucial gap in addressing more complex scenarios such as error detection, for enhancing reasoning capability in complicated settings. To fill this gap, we formally formulate the new task: multimodal error detection, and introduce ErrorRadar, the first benchmark designed to assess MLLMs' capabilities in such a task. ErrorRadar evaluates two sub-tasks: error step identification and error categorization, providing a comprehensive framework for evaluating MLLMs' complex mathematical reasoning ability. It consists of 2,500 high-quality multimodal K-12 mathematical problems, collected from real-world student interactions in an educational organization, with rigorous annotation and rich metadata such as problem type and error category. Through extensive experiments, we evaluated both open-source and closed-source representative MLLMs, benchmarking their performance against educational expert evaluators. Results indicate significant challenges still remain, as GPT-4o with best performance is still around 10% behind human evaluation.