LidarDM: Generative LiDAR Simulation in a Generated World

TL;DR

LidarDM is a new generative model that creates realistic, layout-aware, and temporally coherent LiDAR videos for autonomous driving simulation, using a novel 4D world generation framework with latent diffusion models.

Contribution

The paper introduces LidarDM, the first LiDAR generative model guided by driving scenarios and capable of producing 4D, temporally coherent LiDAR sequences with a novel integrated 4D world generation framework.

Findings

Outperforms existing methods in realism and temporal coherence.

Enables scenario-guided LiDAR generation for autonomous driving.

Serves as a generative world model for perception training.

Abstract

We present LidarDM, a novel LiDAR generative model capable of producing realistic, layout-aware, physically plausible, and temporally coherent LiDAR videos. LidarDM stands out with two unprecedented capabilities in LiDAR generative modeling: (i) LiDAR generation guided by driving scenarios, offering significant potential for autonomous driving simulations, and (ii) 4D LiDAR point cloud generation, enabling the creation of realistic and temporally coherent sequences. At the heart of our model is a novel integrated 4D world generation framework. Specifically, we employ latent diffusion models to generate the 3D scene, combine it with dynamic actors to form the underlying 4D world, and subsequently produce realistic sensory observations within this virtual environment. Our experiments indicate that our approach outperforms competing algorithms in realism, temporal coherency, and layout consistency. We additionally show that LidarDM can be used as a generative world model simulator for training and testing perception models.