UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation

TL;DR

UltraLiDAR introduces a compact, discrete representation for LiDAR data that enables effective scene-level completion, realistic generation, and manipulation, significantly reducing costs and improving perception system performance.

Contribution

The paper presents a novel discrete representation for LiDAR point clouds that enhances densification, generation diversity, and manipulation, outperforming prior methods in realism and effectiveness.

Findings

Densifies sparse LiDAR data to match high-density quality.

Generates diverse, realistic LiDAR point clouds.

Improves downstream perception system performance.

Abstract

LiDAR provides accurate geometric measurements of the 3D world. Unfortunately, dense LiDARs are very expensive and the point clouds captured by low-beam LiDAR are often sparse. To address these issues, we present UltraLiDAR, a data-driven framework for scene-level LiDAR completion, LiDAR generation, and LiDAR manipulation. The crux of UltraLiDAR is a compact, discrete representation that encodes the point cloud's geometric structure, is robust to noise, and is easy to manipulate. We show that by aligning the representation of a sparse point cloud to that of a dense point cloud, we can densify the sparse point clouds as if they were captured by a real high-density LiDAR, drastically reducing the cost. Furthermore, by learning a prior over the discrete codebook, we can generate diverse, realistic LiDAR point clouds for self-driving. We evaluate the effectiveness of UltraLiDAR on sparse-to-dense LiDAR completion and LiDAR generation. Experiments show that densifying real-world point clouds with our approach can significantly improve the performance of downstream perception systems. Compared to prior art on LiDAR generation, our approach generates much more realistic point clouds. According to A/B test, over 98.5\% of the time human participants prefer our results over those of previous methods.