MapTab: Are MLLMs Ready for Multi-Criteria Route Planning in Heterogeneous Graphs?

TL;DR

MapTab is a new multimodal benchmark designed to evaluate large language models' ability to perform multi-criteria route planning using visual and tabular data across diverse global scenarios.

Contribution

The paper introduces MapTab, a comprehensive benchmark for assessing MLLMs' multi-criteria reasoning in route planning tasks with multimodal inputs.

Findings

Current MLLMs struggle with multi-criteria multimodal reasoning.

Multimodal collaboration often underperforms compared to unimodal models under visual limitations.

MapTab offers a challenging testbed for evaluating MLLMs' reasoning capabilities.

Abstract

Systematic evaluation of Multimodal Large Language Models (MLLMs) is crucial for advancing Artificial General Intelligence (AGI). However, existing benchmarks remain insufficient for rigorously assessing their reasoning capabilities under multi-criteria constraints. To bridge this gap, we introduce MapTab, a multimodal benchmark specifically designed to evaluate holistic multi-criteria reasoning in MLLMs via route planning tasks. MapTab requires MLLMs to perceive and ground visual cues from map images alongside route attributes (e.g., Time, Price) from structured tabular data. The benchmark encompasses two scenarios: Metromap, covering metro networks in 160 cities across 52 countries, and Travelmap, depicting 168 representative tourist attractions from 19 countries. In total, MapTab comprises 328 images, 196,800 route planning queries, and 3,936 QA queries, all incorporating 4 key criteria: Time, Price, Comfort, and Reliability. Extensive evaluations across 15 representative MLLMs reveal that current models face substantial challenges in multi-criteria multimodal reasoning. Notably, under conditions of limited visual perception, multimodal collaboration often underperforms compared to unimodal approaches. We believe MapTab provides a challenging and realistic testbed to advance the systematic evaluation of MLLMs. Our code is available at https://github.com/Ziqiao-Shang/MapTab.