LidarPainter: One-Step Away From Any Lidar View To Novel Guidance

TL;DR

LidarPainter is a real-time diffusion model that enhances 3D driving scene reconstruction from sparse LiDAR data, enabling high-fidelity, stylized lane shifts with superior speed and resource efficiency.

Contribution

It introduces a one-step diffusion approach for consistent view synthesis from sparse LiDAR data, outperforming existing methods in speed, quality, and resource use.

Findings

7x faster than StreetCrafter

Uses only one fifth of GPU memory

Supports stylized scene generation with text prompts

Abstract

Dynamic driving scene reconstruction is of great importance in fields like digital twin system and autonomous driving simulation. However, unacceptable degradation occurs when the view deviates from the input trajectory, leading to corrupted background and vehicle models. To improve reconstruction quality on novel trajectory, existing methods are subject to various limitations including inconsistency, deformation, and time consumption. This paper proposes LidarPainter, a one-step diffusion model that recovers consistent driving views from sparse LiDAR condition and artifact-corrupted renderings in real-time, enabling high-fidelity lane shifts in driving scene reconstruction. Extensive experiments show that LidarPainter outperforms state-of-the-art methods in speed, quality and resource efficiency, specifically 7 x faster than StreetCrafter with only one fifth of GPU memory required. LidarPainter also supports stylized generation using text prompts such as "foggy" and "night", allowing for a diverse expansion of the existing asset library.