Harnessing Massive Satellite Imagery with Efficient Masked Image Modeling

TL;DR

This paper introduces a large-scale satellite imagery dataset and an efficient masked image modeling method, significantly improving remote sensing model performance and training efficiency.

Contribution

It presents OpticalRS-13M, a new extensive dataset, and SelectiveMAE, an efficient MIM approach that reduces computational costs in remote sensing pre-training.

Findings

OpticalRS-13M enhances model performance across tasks.

SelectiveMAE doubles training efficiency.

The pipeline scales effectively for remote sensing models.

Abstract

Masked Image Modeling (MIM) has become an essential method for building foundational visual models in remote sensing (RS). However, the limitations in size and diversity of existing RS datasets restrict the ability of MIM methods to learn generalizable representations. Additionally, conventional MIM techniques, which require reconstructing all tokens, introduce unnecessary computational overhead. To address these issues, we present a new pre-training pipeline for RS models, featuring the creation of a large-scale RS dataset and an efficient MIM approach. We curated a high-quality dataset named OpticalRS-13M by collecting publicly available RS datasets and processing them through exclusion, slicing, and deduplication. OpticalRS-13M comprises 13 million optical images covering various RS tasks, such as object detection and pixel segmentation. To enhance efficiency, we propose SelectiveMAE, a pre-training method that dynamically encodes and reconstructs semantically rich patch tokens, thereby reducing the inefficiencies of traditional MIM models caused by redundant background pixels in RS images. Extensive experiments show that OpticalRS-13M significantly improves classification, detection, and segmentation performance, while SelectiveMAE increases training efficiency over 2$\times$ times. This highlights the effectiveness and scalability of our pipeline in developing RS foundational models. The dataset, source code, and trained models will be released at https://github.com/MiliLab/SelectiveMAE.