ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Maps

TL;DR

ReasonMap introduces a new benchmark with transit maps and questions to evaluate multimodal models' visual reasoning, revealing insights into model performance and grounding requirements.

Contribution

The paper presents ReasonMap, a novel high-resolution transit map benchmark with a two-level evaluation pipeline for assessing multimodal large language models' reasoning capabilities.

Findings

Open-source base models outperform reasoning-tuned variants.

Closed-source models show better reasoning after tuning.

Strong visual grounding is essential for high performance.

Abstract

Multimodal large language models (MLLMs) have demonstrated significant progress in semantic scene understanding and text-image alignment, with reasoning variants enhancing performance on more complex tasks involving mathematics and logic. To bridge this gap, we introduce ReasonMap, a novel benchmark specifically designed to evaluate these capabilities. ReasonMap encompasses high-resolution transit maps from 30 cities and includes 1,008 question-answer pairs spanning two question types and three templates. Furthermore, we design a two-level evaluation pipeline that properly assesses answer correctness and quality. Our comprehensive evaluation of 16 popular MLLMs reveals a counterintuitive pattern: among open-source models, base variants outperform their reasoning-tuned counterparts, whereas the opposite trend is observed in closed-source models. Further analysis under the visual-masking setting confirms that strong performance necessitates direct visual grounding, rather than relying solely on language priors. We further establish a training baseline with reinforcement fine-tuning, providing a reference for future exploration. We hope this benchmark study offers new insights into visual reasoning and helps investigate the gap between open- and closed-source models.