Deep Multimodal Fusion for Semantic Segmentation of Remote Sensing Earth Observation Data

TL;DR

This paper introduces a late fusion deep learning model that combines high-resolution aerial imagery and satellite time series data to significantly improve semantic segmentation accuracy in remote sensing applications.

Contribution

It presents a novel dual-branch deep learning framework that effectively fuses spatial and temporal data sources for enhanced land cover segmentation.

Findings

Achieves state-of-the-art results on the FLAIR dataset.

Demonstrates the effectiveness of multi-modality fusion in remote sensing.

Improves segmentation robustness and accuracy.

Abstract

Accurate semantic segmentation of remote sensing imagery is critical for various Earth observation applications, such as land cover mapping, urban planning, and environmental monitoring. However, individual data sources often present limitations for this task. Very High Resolution (VHR) aerial imagery provides rich spatial details but cannot capture temporal information about land cover changes. Conversely, Satellite Image Time Series (SITS) capture temporal dynamics, such as seasonal variations in vegetation, but with limited spatial resolution, making it difficult to distinguish fine-scale objects. This paper proposes a late fusion deep learning model (LF-DLM) for semantic segmentation that leverages the complementary strengths of both VHR aerial imagery and SITS. The proposed model consists of two independent deep learning branches. One branch integrates detailed textures from aerial imagery captured by UNetFormer with a Multi-Axis Vision Transformer (MaxViT) backbone. The other branch captures complex spatio-temporal dynamics from the Sentinel-2 satellite image time series using a U-Net with Temporal Attention Encoder (U-TAE). This approach leads to state-of-the-art results on the FLAIR dataset, a large-scale benchmark for land cover segmentation using multi-source optical imagery. The findings highlight the importance of multi-modality fusion in improving the accuracy and robustness of semantic segmentation in remote sensing applications.