Self-supervised Vision Transformers for Joint SAR-optical Representation Learning

TL;DR

This paper introduces a novel self-supervised learning approach using Vision Transformers for joint SAR-optical data representation, enhancing remote sensing analysis without requiring labeled data.

Contribution

It combines SAR and optical imagery in a unified ViT-based SSL framework, utilizing channel masking and multimodal training to improve representation learning.

Findings

ViT backbones outperform ConvNets in this setting.

The proposed DINO-MM improves multimodal feature learning.

Enhanced performance on BigEarthNet-MM dataset.

Abstract

Self-supervised learning (SSL) has attracted much interest in remote sensing and earth observation due to its ability to learn task-agnostic representations without human annotation. While most of the existing SSL works in remote sensing utilize ConvNet backbones and focus on a single modality, we explore the potential of vision transformers (ViTs) for joint SAR-optical representation learning. Based on DINO, a state-of-the-art SSL algorithm that distills knowledge from two augmented views of an input image, we combine SAR and optical imagery by concatenating all channels to a unified input. Subsequently, we randomly mask out channels of one modality as a data augmentation strategy. While training, the model gets fed optical-only, SAR-only, and SAR-optical image pairs learning both inner- and intra-modality representations. Experimental results employing the BigEarthNet-MM dataset demonstrate the benefits of both, the ViT backbones and the proposed multimodal SSL algorithm DINO-MM.