GEM: Generating LiDAR World Model via Deformable Mamba

TL;DR

GEM introduces a deformable mamba-based generative model for LiDAR world modeling, improving fidelity and dynamic scene understanding for autonomous driving applications.

Contribution

The paper presents a novel deformable mamba architecture tailored for LiDAR data, enabling better spatial-temporal modeling and scene generation capabilities.

Findings

Achieves state-of-the-art performance on multiple benchmarks.

Effectively disentangles static and dynamic features in LiDAR data.

Demonstrates potential for autonomous planning and 'what-if' scenario generation.

Abstract

World models, which simulate environmental dynamics and generate sensor observations, are gaining increasing attention in autonomous driving. However, progress in LiDAR-based world models has lagged behind those built on camera videos or occupancy data, primarily due to two core challenges: the inherent disorder of LiDAR point clouds and the difficulty of distinguishing dynamic objects from static structures. To address these issues, we propose GEM: a Generative LiDAR world model that leverages deformable mamba architecture, significantly improving fidelity and imaginative capability. Specifically, leveraging the structural similarity between sequential laser scanning and Mamba's processing mechanism, we first tokenize LiDAR sweeps into compact representations via a custom LiDAR scene tokenizer. After unsupervised disentanglement of tokenized features via a dynamic-static separator, a tri-path deformable Mamba is introduced to perform selective scanning and adaptive gating fusion over the disentangled features, leading to enhanced spatial-temporal understanding of the world evolution. Optionally, a planner and a BEV layout controller can be integrated to explore the model's capability for autonomous rollout and its potential to generate ``what-if" scenarios. Extensive experiments show that GEM achieves state-of-the-art performances across diverse benchmarks and evaluation settings, demonstrating its superiority and effectiveness. Project page: https://github.com/wuyang98/GEM.