MultiMAE Meets Earth Observation: Pre-training Multi-modal Multi-task Masked Autoencoders for Earth Observation Tasks

TL;DR

This paper introduces MultiMAE, a flexible multi-modal, multi-task masked autoencoder for Earth Observation data, significantly enhancing transfer learning across diverse EO tasks by reconstructing multiple data modalities.

Contribution

It presents a novel multi-modal, multi-task pre-training strategy for EO data that improves transfer learning and handles diverse input configurations without modality-specific models.

Findings

Outperforms state-of-the-art methods on EO classification and segmentation datasets.

Demonstrates robustness and flexibility across various input modalities.

Enables effective transfer learning without modality-specific pre-training.

Abstract

Multi-modal data in Earth Observation (EO) presents a huge opportunity for improving transfer learning capabilities when pre-training deep learning models. Unlike prior work that often overlooks multi-modal EO data, recent methods have started to include it, resulting in more effective pre-training strategies. However, existing approaches commonly face challenges in effectively transferring learning to downstream tasks where the structure of available data differs from that used during pre-training. This paper addresses this limitation by exploring a more flexible multi-modal, multi-task pre-training strategy for EO data. Specifically, we adopt a Multi-modal Multi-task Masked Autoencoder (MultiMAE) that we pre-train by reconstructing diverse input modalities, including spectral, elevation, and segmentation data. The pre-trained model demonstrates robust transfer learning capabilities, outperforming state-of-the-art methods on various EO datasets for classification and segmentation tasks. Our approach exhibits significant flexibility, handling diverse input configurations without requiring modality-specific pre-trained models. Code will be available at: https://github.com/josesosajs/multimae-meets-eo.