Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networks

TL;DR

This paper explores advanced deep learning techniques for semantic labeling of very high resolution multi-modal remote sensing data, focusing on multi-scale approaches and fusion strategies of Lidar and multispectral data.

Contribution

It introduces a multi-scale method and compares early and late fusion techniques for multi-modal remote sensing data, validated on public datasets.

Findings

Late fusion reduces errors from ambiguous data

Early fusion enables better joint-feature learning

Late fusion is more robust to missing data

Abstract

In this work, we investigate various methods to deal with semantic labeling of very high resolution multi-modal remote sensing data. Especially, we study how deep fully convolutional networks can be adapted to deal with multi-modal and multi-scale remote sensing data for semantic labeling. Our contributions are threefold: a) we present an efficient multi-scale approach to leverage both a large spatial context and the high resolution data, b) we investigate early and late fusion of Lidar and multispectral data, c) we validate our methods on two public datasets with state-of-the-art results. Our results indicate that late fusion make it possible to recover errors steaming from ambiguous data, while early fusion allows for better joint-feature learning but at the cost of higher sensitivity to missing data.