Generative AI Empowered LiDAR Point Cloud Generation with Multimodal Transformer

TL;DR

This paper introduces a multimodal transformer-based method to synthesize LiDAR point clouds from images and RADAR data, improving environmental sensing for 6G wireless systems, especially in adverse conditions.

Contribution

It presents a novel framework combining pre-trained encoders and transformers to generate LiDAR data from multimodal inputs, enhancing environmental perception in wireless communication.

Findings

Achieves a mean squared error of 10.3931 in LiDAR synthesis

Successfully captures structures in diverse environments

Enhances environmental sensing for wireless applications

Abstract

Integrated sensing and communications is a key enabler for the 6G wireless communication systems. The multiple sensing modalities will allow the base station to have a more accurate representation of the environment, leading to context-aware communications. Some widely equipped sensors such as cameras and RADAR sensors can provide some environmental perceptions. However, they are not enough to generate precise environmental representations, especially in adverse weather conditions. On the other hand, the LiDAR sensors provide more accurate representations, however, their widespread adoption is hindered by their high cost. This paper proposes a novel approach to enhance the wireless communication systems by synthesizing LiDAR point clouds from images and RADAR data. Specifically, it uses a multimodal transformer architecture and pre-trained encoding models to enable an accurate LiDAR generation. The proposed framework is evaluated on the DeepSense 6G dataset, which is a real-world dataset curated for context-aware wireless applications. Our results demonstrate the efficacy of the proposed approach in accurately generating LiDAR point clouds. We achieve a modified mean squared error of 10.3931. Visual examination of the images indicates that our model can successfully capture the majority of structures present in the LiDAR point cloud for diverse environments. This will enable the base stations to achieve more precise environmental sensing. By integrating LiDAR synthesis with existing sensing modalities, our method can enhance the performance of various wireless applications, including beam and blockage prediction.