RemoteSAM: Towards Segment Anything for Earth Observation

TL;DR

RemoteSAM is a new foundation model for Earth observation that leverages a large-scale dataset and a unified task paradigm to excel across diverse perception tasks with high efficiency.

Contribution

The paper introduces a scalable data engine and a task unification paradigm, enabling a versatile and efficient Earth observation foundation model called RemoteSAM.

Findings

Achieved state-of-the-art results on multiple Earth observation benchmarks.

Created the largest diverse dataset of 270K image-text-mask triplets.

Outperformed existing models like Falcon, GeoChat, and LHRS-Bot.

Abstract

We aim to develop a robust yet flexible visual foundation model for Earth observation. It should possess strong capabilities in recognizing and localizing diverse visual targets while providing compatibility with various input-output interfaces required across different task scenarios. Current systems cannot meet these requirements, as they typically utilize task-specific architecture trained on narrow data domains with limited semantic coverage. Our study addresses these limitations from two aspects: data and modeling. We first introduce an automatic data engine that enjoys significantly better scalability compared to previous human annotation or rule-based approaches. It has enabled us to create the largest dataset of its kind to date, comprising 270K image-text-mask triplets covering an unprecedented range of diverse semantic categories and attribute specifications. Based on this data foundation, we further propose a task unification paradigm that centers around referring expression segmentation. It effectively handles a wide range of vision-centric perception tasks, including classification, detection, segmentation, grounding, etc, using a single model without any task-specific heads. Combining these innovations on data and modeling, we present RemoteSAM, a foundation model that establishes new SoTA on several earth observation perception benchmarks, outperforming other foundation models such as Falcon, GeoChat, and LHRS-Bot with significantly higher efficiency. Models and data are publicly available at https://github.com/1e12Leon/RemoteSAM.