X-ModalNet: A Semi-Supervised Deep Cross-Modal Network for Classification of Remote Sensing Data

TL;DR

X-ModalNet is a semi-supervised deep learning framework that enhances remote sensing data classification by transferring discriminative features across modalities like MSI, SAR, and hyperspectral images, improving accuracy with limited labeled data.

Contribution

The paper introduces X-ModalNet, a novel semi-supervised cross-modal network with modules for label propagation and interactive learning, advancing remote sensing classification with limited annotations.

Findings

Significant accuracy improvements over state-of-the-art methods.

Effective transfer of discriminative features across modalities.

Robust semi-supervised learning with label propagation on high-level features.

Abstract

This paper addresses the problem of semi-supervised transfer learning with limited cross-modality data in remote sensing. A large amount of multi-modal earth observation images, such as multispectral imagery (MSI) or synthetic aperture radar (SAR) data, are openly available on a global scale, enabling parsing global urban scenes through remote sensing imagery. However, their ability in identifying materials (pixel-wise classification) remains limited, due to the noisy collection environment and poor discriminative information as well as limited number of well-annotated training images. To this end, we propose a novel cross-modal deep-learning framework, called X-ModalNet, with three well-designed modules: self-adversarial module, interactive learning module, and label propagation module, by learning to transfer more discriminative information from a small-scale hyperspectral image (HSI) into the classification task using a large-scale MSI or SAR data. Significantly, X-ModalNet generalizes well, owing to propagating labels on an updatable graph constructed by high-level features on the top of the network, yielding semi-supervised cross-modality learning. We evaluate X-ModalNet on two multi-modal remote sensing datasets (HSI-MSI and HSI-SAR) and achieve a significant improvement in comparison with several state-of-the-art methods.