Depth- and Semantics-aware Multi-modal Domain Translation: Generating 3D Panoramic Color Images from LiDAR Point Clouds

TL;DR

This paper introduces TITAN-Next, a novel depth- and semantics-aware generative model that translates LiDAR point clouds into RGB-D images, improving cross-modal translation for autonomous vehicle applications.

Contribution

It presents the first framework leveraging scene semantics for LiDAR-to-camera image translation, enhancing accuracy and practical utility in autonomous driving.

Findings

Outperforms previous models by 23.7% IoU on Semantic-KITTI dataset.

Leverages scene semantics as a mid-level representation.

Demonstrates practical applications in autonomous vehicle safety and data augmentation.

Abstract

This work presents a new depth- and semantics-aware conditional generative model, named TITAN-Next, for cross-domain image-to-image translation in a multi-modal setup between LiDAR and camera sensors. The proposed model leverages scene semantics as a mid-level representation and is able to translate raw LiDAR point clouds to RGB-D camera images by solely relying on semantic scene segments. We claim that this is the first framework of its kind and it has practical applications in autonomous vehicles such as providing a fail-safe mechanism and augmenting available data in the target image domain. The proposed model is evaluated on the large-scale and challenging Semantic-KITTI dataset, and experimental findings show that it considerably outperforms the original TITAN-Net and other strong baselines by 23.7$\%$ margin in terms of IoU.