OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Tasks

TL;DR

OmniEarth introduces a comprehensive benchmark with 28 geospatial tasks for evaluating vision-language models in Earth observation, highlighting current models' limitations in complex remote sensing scenarios.

Contribution

This work presents OmniEarth, the first systematic benchmark for remote sensing vision-language models, covering perception, reasoning, and robustness in diverse geospatial tasks.

Findings

Existing VLMs struggle with geospatially complex tasks.

Benchmark reveals significant gaps in current models' capabilities.

OmniEarth provides a standardized platform for future model development.

Abstract

Vision-Language Models (VLMs) have demonstrated effective perception and reasoning capabilities on general-domain tasks, leading to growing interest in their application to Earth observation. However, a systematic benchmark for comprehensively evaluating remote sensing vision-language models (RSVLMs) remains lacking. To address this gap, we introduce OmniEarth, a benchmark for evaluating RSVLMs under realistic Earth observation scenarios. OmniEarth organizes tasks along three capability dimensions: perception, reasoning, and robustness. It defines 28 fine-grained tasks covering multi-source sensing data and diverse geospatial contexts. The benchmark supports two task formulations: multiple-choice VQA and open-ended VQA. The latter includes pure text outputs for captioning tasks, bounding box outputs for visual grounding tasks, and mask outputs for segmentation tasks. To reduce linguistic bias and examine whether model predictions rely on visual evidence, OmniEarth adopts a blind test protocol and a quintuple semantic consistency requirement. OmniEarth includes 9,275 carefully quality-controlled images, including proprietary satellite imagery from Jilin-1 (JL-1), along with 44,210 manually verified instructions. We conduct a systematic evaluation of contrastive learning-based models, general closed-source and open-source VLMs, as well as RSVLMs. Results show that existing VLMs still struggle with geospatially complex tasks, revealing clear gaps that need to be addressed for remote sensing applications. OmniEarth is publicly available at https://huggingface.co/datasets/sjeeudd/OmniEarth.