GPSBench: Do Large Language Models Understand GPS Coordinates?

TL;DR

GPSBench is a new dataset designed to evaluate large language models' ability to understand and reason about GPS coordinates and geography, revealing current limitations and potential improvements in geospatial reasoning.

Contribution

We introduce GPSBench, a comprehensive dataset for assessing geospatial reasoning in LLMs, and analyze model performance, robustness, and the effects of fine-tuning on GPS coordinate understanding.

Findings

Models struggle with geometric coordinate operations.

Geographic knowledge degrades at finer spatial scales.

Coordinate noise robustness indicates genuine understanding.

Abstract

Large Language Models (LLMs) are increasingly deployed in applications that interact with the physical world, such as navigation, robotics, or mapping, making robust geospatial reasoning a critical capability. Despite that, LLMs' ability to reason about GPS coordinates and real-world geography remains underexplored. We introduce GPSBench, a dataset of 57,800 samples across 17 tasks for evaluating geospatial reasoning in LLMs, spanning geometric coordinate operations (e.g., distance and bearing computation) and reasoning that integrates coordinates with world knowledge. Focusing on intrinsic model capabilities rather than tool use, we evaluate 14 state-of-the-art LLMs and find that GPS reasoning remains challenging, with substantial variation across tasks: models are generally more reliable at real-world geographic reasoning than at geometric computations. Geographic knowledge degrades hierarchically, with strong country-level performance but weak city-level localization, while robustness to coordinate noise suggests genuine coordinate understanding rather than memorization. We further show that GPS-coordinate augmentation can improve in downstream geospatial tasks, and that finetuning induces trade-offs between gains in geometric computation and degradation in world knowledge. Our dataset and reproducible code are available at https://github.com/joey234/gpsbench