MaRVL-QA: A Benchmark for Mathematical Reasoning over Visual Landscapes

TL;DR

MaRVL-QA is a new benchmark designed to evaluate mathematical and spatial reasoning in multimodal models using surface plots, revealing current models' limitations and guiding future improvements.

Contribution

Introduces MaRVL-QA, a benchmark with novel tasks for assessing deep reasoning over visual mathematical landscapes in multimodal models.

Findings

State-of-the-art models perform poorly on the benchmark.

Models tend to rely on superficial heuristics rather than true reasoning.

MaRVL-QA exposes significant gaps in current model capabilities.

Abstract

A key frontier for Multimodal Large Language Models (MLLMs) is the ability to perform deep mathematical and spatial reasoning directly from images, moving beyond their established success in semantic description. Mathematical surface plots provide a rigorous testbed for this capability, as they isolate the task of reasoning from the semantic noise common in natural images. To measure progress on this frontier, we introduce MaRVL-QA (Mathematical Reasoning over Visual Landscapes), a new benchmark designed to quantitatively evaluate these core reasoning skills. The benchmark comprises two novel tasks: Topological Counting, identifying and enumerating features like local maxima; and Transformation Recognition, recognizing applied geometric transformations. Generated from a curated library of functions with rigorous ambiguity filtering, our evaluation on MaRVL-QA reveals that even state-of-the-art MLLMs struggle significantly, often resorting to superficial heuristics instead of robust spatial reasoning. MaRVL-QA provides a challenging new tool for the research community to measure progress, expose model limitations, and guide the development of MLLMs with more profound reasoning abilities.