GeoChain: Multimodal Chain-of-Thought for Geographic Reasoning

TL;DR

GeoChain is a comprehensive benchmark dataset designed to evaluate and improve step-by-step geographic reasoning in multimodal large language models, highlighting current challenges and guiding future advancements.

Contribution

The paper introduces GeoChain, a large-scale multimodal benchmark with detailed reasoning chains and annotations for geographic reasoning tasks in large language models.

Findings

Contemporary MLLMs struggle with visual grounding and accurate localization.

Models exhibit erratic reasoning as task complexity increases.

GeoChain provides a diagnostic tool for advancing geographic reasoning in MLLMs.

Abstract

This paper introduces GeoChain, a large-scale benchmark for evaluating step-by-step geographic reasoning in multimodal large language models (MLLMs). Leveraging 1.46 million Mapillary street-level images, GeoChain pairs each image with a 21-step chain-of-thought (CoT) question sequence (over 30 million Q&A pairs). These sequences guide models from coarse attributes to fine-grained localization across four reasoning categories - visual, spatial, cultural, and precise geolocation - annotated by difficulty. Images are also enriched with semantic segmentation (150 classes) and a visual locatability score. Our benchmarking of contemporary MLLMs (GPT-4.1 variants, Claude 3.7, Gemini 2.5 variants) on a diverse 2,088-image subset reveals consistent challenges: models frequently exhibit weaknesses in visual grounding, display erratic reasoning, and struggle to achieve accurate localization, especially as the reasoning complexity escalates. GeoChain offers a robust diagnostic methodology, critical for fostering significant advancements in complex geographic reasoning within MLLMs.