SatSwinMAE: Efficient Autoencoding for Multiscale Time-series Satellite Imagery

TL;DR

SatSwinMAE introduces an efficient hierarchical autoencoder with temporal modeling for satellite time-series imagery, significantly improving performance on various geospatial tasks by capturing multi-scale spatio-temporal dependencies.

Contribution

It extends SwinMAE with temporal integration using Video Swin Transformer blocks, enhancing transfer learning and achieving state-of-the-art results in satellite image analysis.

Findings

Outperforms existing models in land cover segmentation by 10.4% accuracy

Achieves significant improvements in flood and wildfire mapping tasks

Effectively captures multi-scale spatio-temporal features in satellite data

Abstract

Recent advancements in foundation models have significantly impacted various fields, including natural language processing, computer vision, and multi-modal tasks. One area that stands to benefit greatly is Earth observation, where these models can efficiently process large-scale, unlabeled geospatial data. In this work we extend the SwinMAE model to integrate temporal information for satellite time-series data. The architecture employs a hierarchical 3D Masked Autoencoder (MAE) with Video Swin Transformer blocks to effectively capture multi-scale spatio-temporal dependencies in satellite imagery. To enhance transfer learning, we incorporate both encoder and decoder pretrained weights, along with skip connections to preserve scale-specific information. This forms an architecture similar to SwinUNet with an additional temporal component. Our approach shows significant performance improvements over existing state-of-the-art foundation models for all the evaluated downstream tasks: land cover segmentation, building density prediction, flood mapping, wildfire scar mapping and multi-temporal crop segmentation. Particularly, in the land cover segmentation task of the PhilEO Bench dataset, it outperforms other geospatial foundation models with a 10.4% higher accuracy.