Text2LiDAR: Text-guided LiDAR Point Cloud Generation via Equirectangular Transformer

TL;DR

Text2LiDAR introduces a novel equirectangular transformer architecture for efficient, diverse, and text-controllable LiDAR point cloud generation, addressing data collection challenges in traffic environments.

Contribution

The paper presents the first equirectangular transformer-based model for text-guided LiDAR data synthesis, incorporating a control-signal injector and frequency modulator for high-quality, controllable point cloud generation.

Findings

Outperforms existing methods in uncontrolled and text-controlled scenarios.

Generates diverse LiDAR point clouds aligned with textual descriptions.

Achieves high fidelity and detail in generated point clouds.

Abstract

The complex traffic environment and various weather conditions make the collection of LiDAR data expensive and challenging. Achieving high-quality and controllable LiDAR data generation is urgently needed, controlling with text is a common practice, but there is little research in this field. To this end, we propose Text2LiDAR, the first efficient, diverse, and text-controllable LiDAR data generation model. Specifically, we design an equirectangular transformer architecture, utilizing the designed equirectangular attention to capture LiDAR features in a manner with data characteristics. Then, we design a control-signal embedding injector to efficiently integrate control signals through the global-to-focused attention mechanism. Additionally, we devise a frequency modulator to assist the model in recovering high-frequency details, ensuring the clarity of the generated point cloud. To foster development in the field and optimize text-controlled generation performance, we construct nuLiDARtext which offers diverse text descriptors for 34,149 LiDAR point clouds from 850 scenes. Experiments on uncontrolled and text-controlled generation in various forms on KITTI-360 and nuScenes datasets demonstrate the superiority of our approach.