B4DL: A Benchmark for 4D LiDAR LLM in Spatio-Temporal Understanding

TL;DR

B4DL introduces a new benchmark and model for multimodal large language models to understand 4D LiDAR data in dynamic outdoor environments, enabling spatio-temporal reasoning.

Contribution

It provides the first dataset, benchmark, and model capable of directly processing raw 4D LiDAR data for multimodal understanding.

Findings

The dataset includes rendered 4D LiDAR videos and annotations.

The proposed model effectively bridges 4D LiDAR with language understanding.

Benchmark results demonstrate improved spatio-temporal reasoning capabilities.

Abstract

Understanding dynamic outdoor environments requires capturing complex object interactions and their evolution over time. LiDAR-based 4D point clouds provide precise spatial geometry and rich temporal cues, making them ideal for representing real-world scenes. However, despite their potential, 4D LiDAR remains underexplored in the context of Multimodal Large Language Models (MLLMs) due to the absence of high-quality, modality-specific annotations and the lack of MLLM architectures capable of processing its high-dimensional composition. To address these challenges, we introduce B4DL, a new benchmark specifically designed for training and evaluating MLLMs on 4D LiDAR understanding. In addition, we propose a scalable data generation pipeline and an MLLM model that, for the first time, directly processes raw 4D LiDAR by bridging it with language understanding. Combined with our dataset and benchmark, our model offers a unified solution for spatio-temporal reasoning in dynamic outdoor environments. We provide rendered 4D LiDAR videos, generated dataset, and inference outputs on diverse scenarios at: https://github.com/ccho4702/B4DL