Mx2M: Masked Cross-Modality Modeling in Domain Adaptation for 3D Semantic Segmentation

TL;DR

This paper introduces Mx2M, a novel masked cross-modality modeling approach for 3D semantic segmentation domain adaptation, effectively reducing domain gaps by leveraging self-supervision and dynamic feature matching.

Contribution

The paper proposes Mx2M with cross-modal removal and prediction and a dynamic cross-modal filter, enhancing domain adaptation for 3D segmentation with improved performance.

Findings

Large improvements over previous methods across multiple scenarios.

Effective reduction of domain gap in 3D semantic segmentation.

Robust cross-modal self-supervision mechanism.

Abstract

Existing methods of cross-modal domain adaptation for 3D semantic segmentation predict results only via 2D-3D complementarity that is obtained by cross-modal feature matching. However, as lacking supervision in the target domain, the complementarity is not always reliable. The results are not ideal when the domain gap is large. To solve the problem of lacking supervision, we introduce masked modeling into this task and propose a method Mx2M, which utilizes masked cross-modality modeling to reduce the large domain gap. Our Mx2M contains two components. One is the core solution, cross-modal removal and prediction (xMRP), which makes the Mx2M adapt to various scenarios and provides cross-modal self-supervision. The other is a new way of cross-modal feature matching, the dynamic cross-modal filter (DxMF) that ensures the whole method dynamically uses more suitable 2D-3D complementarity. Evaluation of the Mx2M on three DA scenarios, including Day/Night, USA/Singapore, and A2D2/SemanticKITTI, brings large improvements over previous methods on many metrics.