GeoRC: A Benchmark for Geolocation Reasoning Chains

TL;DR

GeoRC is a new benchmark sourced from GeoGuessr experts that evaluates the ability of vision language models to produce detailed, auditable geolocation reasoning chains, revealing current limitations of VLMs in fine-grained visual attribute extraction.

Contribution

This paper introduces GeoRC, the first benchmark for geolocation reasoning chains from expert sources, and evaluates VLMs' reasoning capabilities against human experts.

Findings

Large closed-source VLMs like Gemini and GPT-5 rival humans in location prediction.

Open-weight VLMs like Llama and Qwen perform poorly, only slightly better than random guessing.

VLMs struggle with extracting fine-grained visual attributes from high-resolution images.

Abstract

Vision Language Models (VLMs) are good at recognizing the global location of a photograph -- their geolocation prediction accuracy rivals the best human experts. But many VLMs are startlingly bad at \textit{explaining} which image evidence led to their prediction, even when their location prediction is correct. In this paper, we introduce GeoRC, the first benchmark for geolocation reasoning chains sourced directly from Champion-tier GeoGuessr experts, including the reigning world champion. This benchmark consists of 800 ``ground truth'' reasoning chains across 500 query scenes from GeoGuessr maps, with expert chains addressing hundreds of different discriminative attributes, such as soil properties, architecture, and license plate shapes. We evaluate LLM-as-a-judge and VLM-as-a-judge strategies for scoring VLM-generated reasoning chains against our expert reasoning chains and find that Qwen 3 LLM-as-a-judge correlates best with human-expert scoring. Our benchmark reveals that while large, closed-source VLMs such as Gemini and GPT 5 rival human experts at predicting locations, they still lag behind human experts when it comes to producing auditable reasoning chains. Small open-weight VLMs such as Llama and Qwen catastrophically fail on our benchmark -- they perform only slightly better than a baseline in which an LLM hallucinates a reasoning chain with oracle knowledge of the photo location but \textit{no visual information at all}. We believe the gap between human experts and VLMs on this task points to VLM limitations at extracting fine-grained visual attributes from high resolution images. We open source our benchmark for the community to use.