Cross-modal & Cross-domain Learning for Unsupervised LiDAR Semantic Segmentation

TL;DR

This paper introduces CoMoDaL, a novel method for unsupervised LiDAR semantic segmentation that leverages cross-modal and cross-domain learning to eliminate the need for source 3D data, using only 2D images with annotations and unannotated paired data.

Contribution

It proposes a new setting for 3D LiDAR segmentation that removes the requirement for source 3D data and introduces a cross-modal and cross-domain learning framework.

Findings

Achieves LiDAR segmentation without source 3D supervision

Effective cross-modal distillation between 2D images and 3D LiDAR data

Outperforms baseline methods in unsupervised setting

Abstract

In recent years, cross-modal domain adaptation has been studied on the paired 2D image and 3D LiDAR data to ease the labeling costs for 3D LiDAR semantic segmentation (3DLSS) in the target domain. However, in such a setting the paired 2D and 3D data in the source domain are still collected with additional effort. Since the 2D-3D projections can enable the 3D model to learn semantic information from the 2D counterpart, we ask whether we could further remove the need of source 3D data and only rely on the source 2D images. To answer it, this paper studies a new 3DLSS setting where a 2D dataset (source) with semantic annotations and a paired but unannotated 2D image and 3D LiDAR data (target) are available. To achieve 3DLSS in this scenario, we propose Cross-Modal and Cross-Domain Learning (CoMoDaL). Specifically, our CoMoDaL aims at modeling 1) inter-modal cross-domain distillation between the unpaired source 2D image and target 3D LiDAR data, and 2) the intra-domain cross-modal guidance between the target 2D image and 3D LiDAR data pair. In CoMoDaL, we propose to apply several constraints, such as point-to-pixel and prototype-to-pixel alignments, to associate the semantics in different modalities and domains by constructing mixed samples in two modalities. The experimental results on several datasets show that in the proposed setting, the developed CoMoDaL can achieve segmentation without the supervision of labeled LiDAR data. Ablations are also conducted to provide more analysis. Code will be available publicly.