Fake it, Mix it, Segment it: Bridging the Domain Gap Between Lidar Sensors

TL;DR

This paper introduces a novel lidar domain adaptation method that recreates scenes in different sensor structures and mixes data to significantly improve segmentation performance across diverse lidar sensors and datasets.

Contribution

The authors propose a new scene recreation and data mixing approach for lidar domain adaptation, outperforming existing methods in unsupervised and semi-supervised settings.

Findings

15.2 mIoU improvement from nuScenes to SemanticKITTI

48.3 mIoU gain in semi-supervised adaptation

Successful application to unlabeled Velodyne Alpha Prime and InnovizTwo datasets

Abstract

Segmentation of lidar data is a task that provides rich, point-wise information about the environment of robots or autonomous vehicles. Currently best performing neural networks for lidar segmentation are fine-tuned to specific datasets. Switching the lidar sensor without retraining on a big set of annotated data from the new sensor creates a domain shift, which causes the network performance to drop drastically. In this work we propose a new method for lidar domain adaption, in which we use annotated panoptic lidar datasets and recreate the recorded scenes in the structure of a different lidar sensor. We narrow the domain gap to the target data by recreating panoptic data from one domain in another and mixing the generated data with parts of (pseudo) labeled target domain data. Our method improves the nuScenes to SemanticKITTI unsupervised domain adaptation performance by 15.2 mean Intersection over Union points (mIoU) and by 48.3 mIoU in our semi-supervised approach. We demonstrate a similar improvement for the SemanticKITTI to nuScenes domain adaptation by 21.8 mIoU and 51.5 mIoU, respectively. We compare our method with two state of the art approaches for semantic lidar segmentation domain adaptation with a significant improvement for unsupervised and semi-supervised domain adaptation. Furthermore we successfully apply our proposed method to two entirely unlabeled datasets of two state of the art lidar sensors Velodyne Alpha Prime and InnovizTwo, and train well performing semantic segmentation networks for both.