XD-MAP: Cross-Modal Domain Adaptation via Semantic Parametric Maps for Scalable Training Data Generation

TL;DR

XD-MAP introduces a novel cross-modal domain adaptation method that transfers semantic knowledge from images to LiDAR, significantly improving segmentation performance without manual labeling.

Contribution

The paper presents XD-MAP, a new approach that enables sensor-agnostic semantic transfer, extending perception capabilities from camera images to LiDAR without requiring sensor overlap.

Findings

Outperforms baseline by +19.5 mIoU in 2D semantic segmentation

Achieves +19.5 PQth in 2D panoptic segmentation

Improves 3D semantic segmentation by +32.3 mIoU

Abstract

Until open-world foundation models match the performance of specialized approaches, deep learning systems remain dependent on task- and sensor-specific data availability. To bridge the gap between available datasets and deployment domains, domain adaptation strategies are widely used. In this work, we propose XD-MAP, a novel approach to transfer sensor-specific knowledge from an image dataset to LiDAR, an entirely different sensing domain. Our method leverages detections on camera images to create a semantic parametric map. The map elements are modeled to produce pseudo labels in the target domain without any manual annotation effort. Unlike previous domain transfer approaches, our method does not require direct overlap between sensors and enables extending the angular perception range from a front-view camera to a full 360{\deg} view. On our large-scale road feature dataset, XD-MAP outperforms single shot baseline approaches by +19.5 mIoU for 2D semantic segmentation, +19.5 PQth for 2D panoptic segmentation, and +32.3 mIoU in 3D semantic segmentation. The results demonstrate the effectiveness of our approach achieving strong performance on LiDAR data without any manual labeling.