Heterogeneous-Modal Unsupervised Domain Adaptation via Latent Space Bridging

TL;DR

This paper introduces a novel framework called Latent Space Bridging for unsupervised domain adaptation across different modalities, enabling effective knowledge transfer by leveraging a bridge domain with unlabeled data, and demonstrates state-of-the-art results in semantic segmentation.

Contribution

The paper proposes a new setting HMUDA and a specialized framework LSB for semantic segmentation, addressing the challenge of cross-modal domain adaptation.

Findings

LSB achieves state-of-the-art performance on six benchmark datasets.

Latent Space Bridging effectively aligns representations across heterogeneous modalities.

The dual-branch architecture with feature and domain alignment losses improves adaptation quality.

Abstract

Unsupervised domain adaptation (UDA) methods effectively bridge domain gaps but become struggled when the source and target domains belong to entirely distinct modalities. To address this limitation, we propose a novel setting called Heterogeneous-Modal Unsupervised Domain Adaptation (HMUDA), which enables knowledge transfer between completely different modalities by leveraging a bridge domain containing unlabeled samples from both modalities. To learn under the HMUDA setting, we propose Latent Space Bridging (LSB), a specialized framework designed for the semantic segmentation task. Specifically, LSB utilizes a dual-branch architecture, incorporating a feature consistency loss to align representations across modalities and a domain alignment loss to reduce discrepancies between class centroids across domains. Extensive experiments conducted on six benchmark datasets demonstrate that LSB achieves state-of-the-art performance.