Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks

TL;DR

This paper presents a deep learning framework for semantic segmentation of Earth Observation images, combining multi-scale analysis and multimodal data fusion to improve accuracy over existing methods.

Contribution

It introduces a transfer learning approach for remote sensing, a multi-kernel convolutional layer for multi-scale prediction aggregation, and residual correction for multimodal data fusion.

Findings

Achieved state-of-the-art accuracy on ISPRS Vaihingen dataset

Demonstrated effective transfer from generic images to remote sensing data

Enhanced multi-modal data integration with residual correction

Abstract

This work investigates the use of deep fully convolutional neural networks (DFCNN) for pixel-wise scene labeling of Earth Observation images. Especially, we train a variant of the SegNet architecture on remote sensing data over an urban area and study different strategies for performing accurate semantic segmentation. Our contributions are the following: 1) we transfer efficiently a DFCNN from generic everyday images to remote sensing images; 2) we introduce a multi-kernel convolutional layer for fast aggregation of predictions at multiple scales; 3) we perform data fusion from heterogeneous sensors (optical and laser) using residual correction. Our framework improves state-of-the-art accuracy on the ISPRS Vaihingen 2D Semantic Labeling dataset.