xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation

TL;DR

xMUDA introduces a novel cross-modal unsupervised domain adaptation method for 3D semantic segmentation that leverages mutual learning between 2D images and 3D point clouds, significantly improving performance across various domain shifts.

Contribution

The paper proposes a cross-modal UDA framework that enables 2D and 3D modalities to learn from each other through mutual mimicking, addressing heterogeneity and domain shift challenges.

Findings

xMUDA outperforms uni-modal UDA methods on multiple domain shift scenarios.

Mutual mimicking improves segmentation accuracy across modalities.

The approach is complementary to existing state-of-the-art UDA techniques.

Abstract

Unsupervised Domain Adaptation (UDA) is crucial to tackle the lack of annotations in a new domain. There are many multi-modal datasets, but most UDA approaches are uni-modal. In this work, we explore how to learn from multi-modality and propose cross-modal UDA (xMUDA) where we assume the presence of 2D images and 3D point clouds for 3D semantic segmentation. This is challenging as the two input spaces are heterogeneous and can be impacted differently by domain shift. In xMUDA, modalities learn from each other through mutual mimicking, disentangled from the segmentation objective, to prevent the stronger modality from adopting false predictions from the weaker one. We evaluate on new UDA scenarios including day-to-night, country-to-country and dataset-to-dataset, leveraging recent autonomous driving datasets. xMUDA brings large improvements over uni-modal UDA on all tested scenarios, and is complementary to state-of-the-art UDA techniques. Code is available at https://github.com/valeoai/xmuda.