NeighborMAE: Exploiting Spatial Dependencies between Neighboring Earth Observation Images in Masked Autoencoders Pretraining

TL;DR

NeighborMAE introduces a novel self-supervised learning approach that leverages spatial dependencies between neighboring Earth Observation images through joint reconstruction, significantly improving representation learning for remote sensing tasks.

Contribution

It is the first to explicitly incorporate spatial dependencies between neighboring images into Masked Autoencoders for Earth Observation data.

Findings

NeighborMAE outperforms existing methods on multiple datasets.

Dynamic mask ratio and loss weight improve learning difficulty.

Spatial dependencies enhance representation quality.

Abstract

Masked Image Modeling has been one of the most popular self-supervised learning paradigms to learn representations from large-scale, unlabeled Earth Observation images. While incorporating multi-modal and multi-temporal Earth Observation data into Masked Image Modeling has been widely explored, the spatial dependencies between images captured from neighboring areas remains largely overlooked. Since the Earth's surface is continuous, neighboring images are highly related and offer rich contextual information for self-supervised learning. To close this gap, we propose NeighborMAE, which learns spatial dependencies by joint reconstruction of neighboring Earth Observation images. To ensure that the reconstruction remains challenging, we leverage a heuristic strategy to dynamically adjust the mask ratio and the pixel-level loss weight. Experimental results across various pretraining datasets and downstream tasks show that NeighborMAE significantly outperforms existing baselines, underscoring the value of neighboring images in Masked Image Modeling for Earth Observation and the efficacy of our designs.