MapIQ: Evaluating Multimodal Large Language Models for Map Question Answering

TL;DR

This paper introduces MapIQ, a comprehensive benchmark dataset for evaluating multimodal large language models on map question answering across various map types and themes, highlighting model robustness and sensitivity to design changes.

Contribution

The paper presents MapIQ, a new large-scale dataset and evaluation framework for assessing MLLMs on diverse map visual question answering tasks and map design variations.

Findings

MLLMs show varying performance across map types and themes.

Map design changes significantly affect MLLM responses.

Humans outperform MLLMs in map question answering tasks.

Abstract

Recent advancements in multimodal large language models (MLLMs) have driven researchers to explore how well these models read data visualizations, e.g., bar charts, scatter plots. More recently, attention has shifted to visual question answering with maps (Map-VQA). However, Map-VQA research has primarily focused on choropleth maps, which cover only a limited range of thematic categories and visual analytical tasks. To address these gaps, we introduce MapIQ, a benchmark dataset comprising 14,706 question-answer pairs across three map types: choropleth maps, cartograms, and proportional symbol maps spanning topics from six distinct themes (e.g., housing, crime). We evaluate multiple MLLMs using six visual analytical tasks, comparing their performance against one another and a human baseline. An additional experiment examining the impact of map design changes (e.g., altered color schemes, modified legend designs, and removal of map elements) provides insights into the robustness and sensitivity of MLLMs, their reliance on internal geographic knowledge, and potential avenues for improving Map-VQA performance.