MathScape: Benchmarking Multimodal Large Language Models in Real-World Mathematical Contexts

TL;DR

MathScape is a new benchmark designed to evaluate multimodal large language models' ability to perform mathematical reasoning in realistic, real-world contexts using images and problems that reflect practical educational scenarios.

Contribution

We introduce MathScape, a benchmark with real-world images and math problems, to better assess MLLMs' reasoning capabilities beyond synthetic data, revealing current model limitations.

Findings

State-of-the-art models underperform on real-world math tasks

Performance on synthetic images does not translate to real-world scenarios

Current models lag behind human reasoning in practical math contexts

Abstract

With the rapid progress of Multimodal LLMs, evaluating their mathematical reasoning capabilities has become an increasingly important research direction. In particular, visual-textual mathematical reasoning serves as a key indicator of an MLLM's ability to comprehend and solve complex, multi-step quantitative problems. While existing benchmarks such as MathVista and MathVerse have advanced the evaluation of multimodal math proficiency, they primarily rely on digitally rendered content and fall short in capturing the complexity of real-world scenarios. To bridge this gap, we introduce MathScape, a novel benchmark focused on assessing MLLMs' reasoning ability in realistic mathematical contexts. MathScape comprises 1,369 high-quality math problems paired with human-captured real-world images, closely reflecting the challenges encountered in practical educational settings. We conduct a thorough multi-dimensional evaluation across nine leading closed-source MLLMs, three open-source MLLMs with over 20 billion parameters, and seven smaller-scale MLLMs. Our results show that even state-of-the-art models struggle with real-world math tasks, lagging behind human performance, highlighting critical limitations in current model capabilities. Moreover, we find that strong performance on synthetic or digitally rendered images does not guarantee similar effectiveness on real-world tasks. This underscores the necessity of MathScape in the next stage of multimodal mathematical reasoning.