REOBench: Benchmarking Robustness of Earth Observation Foundation Models

TL;DR

REOBench is a comprehensive benchmark that evaluates the robustness of Earth observation foundation models against various real-world image corruptions across multiple tasks, revealing significant vulnerabilities and guiding future improvements.

Contribution

This work introduces REOBench, the first benchmark specifically designed to assess the robustness of Earth observation foundation models under diverse corruptions.

Findings

Models degrade significantly under input corruptions.

Robustness varies across tasks, architectures, and corruption types.

Vision-language models exhibit enhanced robustness.

Abstract

Earth observation foundation models have shown strong generalization across multiple Earth observation tasks, but their robustness under real-world perturbations remains underexplored. To bridge this gap, we introduce REOBench, the first comprehensive benchmark for evaluating the robustness of Earth observation foundation models across six tasks and twelve types of image corruptions, including both appearance-based and geometric perturbations. To ensure realistic and fine-grained evaluation, our benchmark focuses on high-resolution optical remote sensing images, which are widely used in critical applications such as urban planning and disaster response. We conduct a systematic evaluation of a broad range of models trained using masked image modeling, contrastive learning, and vision-language pre-training paradigms. Our results reveal that (1) existing Earth observation foundation models experience significant performance degradation when exposed to input corruptions. (2) The severity of degradation varies across tasks, model architectures, backbone sizes, and types of corruption, with performance drop varying from less than 1% to over 20%. (3) Vision-language models show enhanced robustness, particularly in multimodal tasks. REOBench underscores the vulnerability of current Earth observation foundation models to real-world corruptions and provides actionable insights for developing more robust and reliable models. Code and data are publicly available at https://github.com/lx709/REOBench.