SatMamba: Development of Foundation Models for Remote Sensing Imagery Using State Space Models

TL;DR

SatMamba introduces a novel foundation model for remote sensing imagery that combines masked autoencoders with State Space Models, achieving linear computational scaling and improved efficiency for processing complex Earth observation data.

Contribution

The paper presents SatMamba, a new pretraining framework that integrates State Space Models with masked autoencoders, addressing ViT scalability issues in remote sensing applications.

Findings

Achieves linear computational scaling with input length.

Demonstrates superior performance on various remote sensing tasks.

Enables efficient processing of multispectral and multitemporal data.

Abstract

Foundation models refer to deep learning models pretrained on large unlabeled datasets through self-supervised algorithms. In the Earth science and remote sensing communities, there is growing interest in transforming the use of Earth observation data, including satellite and aerial imagery, through foundation models. Various foundation models have been developed for remote sensing, such as those for multispectral, high-resolution, and hyperspectral images, and have demonstrated superior performance on various downstream tasks compared to traditional supervised models. These models are evolving rapidly, with capabilities to handle multispectral, multitemporal, and multisensor data. Most studies use masked autoencoders in combination with Vision Transformers (ViTs) as the backbone for pretraining. While the models showed promising performance, ViTs face challenges, such as quadratic computational scaling with input length, which may limit performance on multiband and multitemporal data with long sequences. This research aims to address these challenges by proposing SatMamba, a new pretraining framework that combines masked autoencoders with State Space Model, offering linear computational scaling. Experiments on high-resolution imagery across various downstream tasks show promising results, paving the way for more efficient foundation models and unlocking the full potential of Earth observation data. The source code is available in https://github.com/mdchuc/HRSFM.