LiDARDraft: Generating LiDAR Point Cloud from Versatile Inputs

TL;DR

LiDARDraft introduces a novel method for generating high-quality, controllable LiDAR point clouds from various user inputs by leveraging 3D layouts and a rangemap-based ControlNet, enhancing autonomous driving simulation capabilities.

Contribution

The paper presents a new approach that unifies different input modalities into 3D layouts to improve controllability and quality in LiDAR point cloud generation.

Findings

Achieves high-quality, controllable LiDAR point clouds from diverse inputs.

Enables simulation from scratch using textual, visual, or sketch inputs.

Demonstrates superior performance over previous methods.

Abstract

Generating realistic and diverse LiDAR point clouds is crucial for autonomous driving simulation. Although previous methods achieve LiDAR point cloud generation from user inputs, they struggle to attain high-quality results while enabling versatile controllability, due to the imbalance between the complex distribution of LiDAR point clouds and the simple control signals. To address the limitation, we propose LiDARDraft, which utilizes the 3D layout to build a bridge between versatile conditional signals and LiDAR point clouds. The 3D layout can be trivially generated from various user inputs such as textual descriptions and images. Specifically, we represent text, images, and point clouds as unified 3D layouts, which are further transformed into semantic and depth control signals. Then, we employ a rangemap-based ControlNet to guide LiDAR point cloud generation. This pixel-level alignment approach demonstrates excellent performance in controllable LiDAR point clouds generation, enabling "simulation from scratch", allowing self-driving environments to be created from arbitrary textual descriptions, images and sketches.