Semantics-aware Multi-modal Domain Translation:From LiDAR Point Clouds to Panoramic Color Images

TL;DR

This paper introduces a novel semantics-aware framework that translates LiDAR point clouds into panoramic color images using semantic segmentation and generative adversarial networks, enabling cross-modal scene synthesis.

Contribution

It is the first to synthesize panoramic images from LiDAR data solely based on semantic information, advancing cross-modal translation techniques.

Findings

Outperforms baseline models on SemanticKitti dataset

Effective semantic segmentation transfer between modalities

High-quality panoramic scene synthesis

Abstract

In this work, we present a simple yet effective framework to address the domain translation problem between different sensor modalities with unique data formats. By relying only on the semantics of the scene, our modular generative framework can, for the first time, synthesize a panoramic color image from a given full 3D LiDAR point cloud. The framework starts with semantic segmentation of the point cloud, which is initially projected onto a spherical surface. The same semantic segmentation is applied to the corresponding camera image. Next, our new conditional generative model adversarially learns to translate the predicted LiDAR segment maps to the camera image counterparts. Finally, generated image segments are processed to render the panoramic scene images. We provide a thorough quantitative evaluation on the SemanticKitti dataset and show that our proposed framework outperforms other strong baseline models. Our source code is available at https://github.com/halmstad-University/TITAN-NET