OpenEarthSensing: Large-Scale Fine-Grained Benchmark for Open-World Remote Sensing

TL;DR

OpenEarthSensing (OES) is a comprehensive large-scale benchmark designed to evaluate open-world remote sensing models across diverse tasks, categories, and data domains, addressing the lack of such benchmarks in the field.

Contribution

The paper introduces OES, a novel large-scale, fine-grained benchmark with 189 categories and multiple data domains for evaluating open-world remote sensing models.

Findings

Existing models struggle with diverse semantic and covariate shifts.

OES provides a challenging testbed for evaluating generalization.

Baseline methods show significant room for improvement.

Abstract

The advancement of remote sensing, including satellite systems, facilitates the continuous acquisition of remote sensing imagery globally, introducing novel challenges for achieving open-world tasks. Deployed models need to continuously adjust to a constant influx of new data, which frequently exhibits diverse shifts from the data encountered during the training phase. To effectively handle the new data, models are required to detect semantic shifts, adapt to covariate shifts, and continuously update their parameters without forgetting learned knowledge, as has been considered in works on a variety of open-world tasks. However, existing studies are typically conducted within a single dataset to simulate realistic conditions, with a lack of large-scale benchmarks capable of evaluating multiple open-world tasks. In this paper, we introduce \textbf{OpenEarthSensing (OES)}, a large-scale fine-grained benchmark for open-world remote sensing. OES includes 189 scene and object categories, covering the vast majority of potential semantic shifts that may occur in the real world. Additionally, to provide a more comprehensive testbed for evaluating the generalization performance, OES encompasses five data domains with significant covariate shifts, including two RGB satellite domains, one RGB aerial domain, one multispectral RGB domain, and one infrared domain. We evaluate the baselines and existing methods for diverse tasks on OES, demonstrating that it serves as a meaningful and challenging benchmark for open-world remote sensing. The proposed dataset OES is available at https://haiv-lab.github.io/OES.