MoPA: Multi-Modal Prior Aided Domain Adaptation for 3D Semantic Segmentation

TL;DR

This paper introduces MoPA, a novel multi-modal domain adaptation method that enhances 3D semantic segmentation, especially for rare objects, by using prior object insertion and 2D semantic masks to address class imbalance.

Contribution

MoPA proposes Valid Ground-based Insertion and SAM consistency loss to improve rare object segmentation in multi-modal unsupervised domain adaptation.

Findings

Achieves state-of-the-art results on MM-UDA benchmark.

Effectively improves rare object segmentation performance.

Utilizes prior object insertion and 2D semantic masks for better adaptation.

Abstract

Multi-modal unsupervised domain adaptation (MM-UDA) for 3D semantic segmentation is a practical solution to embed semantic understanding in autonomous systems without expensive point-wise annotations. While previous MM-UDA methods can achieve overall improvement, they suffer from significant class-imbalanced performance, restricting their adoption in real applications. This imbalanced performance is mainly caused by: 1) self-training with imbalanced data and 2) the lack of pixel-wise 2D supervision signals. In this work, we propose Multi-modal Prior Aided (MoPA) domain adaptation to improve the performance of rare objects. Specifically, we develop Valid Ground-based Insertion (VGI) to rectify the imbalance supervision signals by inserting prior rare objects collected from the wild while avoiding introducing artificial artifacts that lead to trivial solutions. Meanwhile, our SAM consistency loss leverages the 2D prior semantic masks from SAM as pixel-wise supervision signals to encourage consistent predictions for each object in the semantic mask. The knowledge learned from modal-specific prior is then shared across modalities to achieve better rare object segmentation. Extensive experiments show that our method achieves state-of-the-art performance on the challenging MM-UDA benchmark. Code will be available at https://github.com/AronCao49/MoPA.