SMARTIES: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images

TL;DR

SMARTIES is a versatile foundation model that enables flexible multi-sensor remote sensing data processing by projecting diverse sensor data into a shared spectrum-aware space, improving generalization and scalability.

Contribution

The paper introduces SMARTIES, a unified transformer-based model that handles diverse remote sensing sensors without sensor-specific training, advancing multi-sensor data integration.

Findings

Outperforms sensor-specific models on various tasks

Enables arbitrary sensor combinations during training and inference

Achieves sensor-agnostic representations through cross-sensor token mixup

Abstract

From optical sensors to microwave radars, leveraging the complementary strengths of remote sensing (RS) sensors is crucial for achieving dense spatio-temporal monitoring of our planet. In contrast, recent deep learning models, whether task-specific or foundational, are often specific to single sensors or to fixed combinations: adapting such models to different sensory inputs requires both architectural changes and re-training, limiting scalability and generalization across multiple RS sensors. On the contrary, a single model able to modulate its feature representations to accept diverse sensors as input would pave the way to agile and flexible multi-sensor RS data processing. To address this, we introduce SMARTIES, a generic and versatile foundation model lifting sensor-specific/dependent efforts and enabling scalability and generalization to diverse RS sensors: SMARTIES projects data from heterogeneous sensors into a shared spectrum-aware space, enabling the use of arbitrary combinations of bands both for training and inference. To obtain sensor-agnostic representations, we train a single, unified transformer model reconstructing masked multi-sensor data with cross-sensor token mixup. On both single- and multi-modal tasks across diverse sensors, SMARTIES outperforms previous models that rely on sensor-specific pretraining. Our code and pretrained models are available at https://gsumbul.github.io/SMARTIES.