Shared Manifold Learning Using a Triplet Network for Multiple Sensor Translation and Fusion with Missing Data

TL;DR

This paper introduces CoMMANet, a contrastive learning framework that aligns heterogeneous sensor data into a shared manifold for improved classification and sensor translation, especially with missing data scenarios.

Contribution

The paper presents a novel multimodal triplet autoencoder architecture that effectively aligns and fuses data from different sensors into a discriminative shared space, enabling robust classification and sensor translation.

Findings

Achieved 94.3% accuracy on MUUFL dataset.

Outperformed state-of-the-art methods on benchmark datasets.

Demonstrated effective sensor translation with missing data.

Abstract

Heterogeneous data fusion can enhance the robustness and accuracy of an algorithm on a given task. However, due to the difference in various modalities, aligning the sensors and embedding their information into discriminative and compact representations is challenging. In this paper, we propose a Contrastive learning based MultiModal Alignment Network (CoMMANet) to align data from different sensors into a shared and discriminative manifold where class information is preserved. The proposed architecture uses a multimodal triplet autoencoder to cluster the latent space in such a way that samples of the same classes from each heterogeneous modality are mapped close to each other. Since all the modalities exist in a shared manifold, a unified classification framework is proposed. The resulting latent space representations are fused to perform more robust and accurate classification. In a missing sensor scenario, the latent space of one sensor is easily and efficiently predicted using another sensor's latent space, thereby allowing sensor translation. We conducted extensive experiments on a manually labeled multimodal dataset containing hyperspectral data from AVIRIS-NG and NEON, and LiDAR (light detection and ranging) data from NEON. Lastly, the model is validated on two benchmark datasets: Berlin Dataset (hyperspectral and synthetic aperture radar) and MUUFL Gulfport Dataset (hyperspectral and LiDAR). A comparison made with other methods demonstrates the superiority of this method. We achieved a mean overall accuracy of 94.3% on the MUUFL dataset and the best overall accuracy of 71.26% on the Berlin dataset, which is better than other state-of-the-art approaches.