Learning Shared Cross-modality Representation Using Multispectral-LiDAR and Hyperspectral Data

TL;DR

This paper introduces a method to learn a shared feature space across multispectral-LiDAR and hyperspectral data, enabling effective cross-modality representation even when some modalities are missing during testing.

Contribution

The proposed approach learns a shared latent space for multi-modalities during training, improving cross-modality generalization and feature discrimination, especially in the absence of certain modalities at test time.

Findings

Outperforms several baseline methods on the 2018 IEEE GRSS dataset.

Effectively projects out-of-sample data onto the shared space.

Enhances feature discrimination through a latent subspace connection.

Abstract

Due to the ever-growing diversity of the data source, multi-modality feature learning has attracted more and more attention. However, most of these methods are designed by jointly learning feature representation from multi-modalities that exist in both training and test sets, yet they are less investigated in absence of certain modality in the test phase. To this end, in this letter, we propose to learn a shared feature space across multi-modalities in the training process. By this way, the out-of-sample from any of multi-modalities can be directly projected onto the learned space for a more effective cross-modality representation. More significantly, the shared space is regarded as a latent subspace in our proposed method, which connects the original multi-modal samples with label information to further improve the feature discrimination. Experiments are conducted on the multispectral-Lidar and hyperspectral dataset provided by the 2018 IEEE GRSS Data Fusion Contest to demonstrate the effectiveness and superiority of the proposed method in comparison with several popular baselines.