GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AI

TL;DR

GEO-Bench-2 introduces a comprehensive, standardized evaluation framework for Geospatial Foundation Models, enabling nuanced performance assessment across diverse tasks and datasets, and highlighting the importance of task-specific model selection.

Contribution

It provides a flexible, capability-based benchmarking protocol and a diverse dataset suite, facilitating fair comparison and research into model adaptation strategies for GeoFMs.

Findings

No single model dominates across all tasks

Pretraining on natural images benefits high-resolution tasks

EO-specific models excel in multispectral applications

Abstract

Geospatial Foundation Models (GeoFMs) are transforming Earth Observation (EO), but evaluation lacks standardized protocols. GEO-Bench-2 addresses this with a comprehensive framework spanning classification, segmentation, regression, object detection, and instance segmentation across 19 permissively-licensed datasets. We introduce ''capability'' groups to rank models on datasets that share common characteristics (e.g., resolution, bands, temporality). This enables users to identify which models excel in each capability and determine which areas need improvement in future work. To support both fair comparison and methodological innovation, we define a prescriptive yet flexible evaluation protocol. This not only ensures consistency in benchmarking but also facilitates research into model adaptation strategies, a key and open challenge in advancing GeoFMs for downstream tasks. Our experiments show that no single model dominates across all tasks, confirming the specificity of the choices made during architecture design and pretraining. While models pretrained on natural images (ConvNext ImageNet, DINO V3) excel on high-resolution tasks, EO-specific models (TerraMind, Prithvi, and Clay) outperform them on multispectral applications such as agriculture and disaster response. These findings demonstrate that optimal model choice depends on task requirements, data modalities, and constraints. This shows that the goal of a single GeoFM model that performs well across all tasks remains open for future research. GEO-Bench-2 enables informed, reproducible GeoFM evaluation tailored to specific use cases. Code, data, and leaderboard for GEO-Bench-2 are publicly released under a permissive license.