MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusion

TL;DR

MMDrive is a novel multimodal framework that extends scene understanding in autonomous driving from 2D images to 3D by integrating occupancy maps, LiDAR, and text, significantly improving reasoning capabilities.

Contribution

It introduces adaptive cross-modal fusion and key information extraction components, enabling 3D scene understanding beyond traditional image-based models.

Findings

Achieves state-of-the-art performance on DriveLM and NuScenes-QA benchmarks.

Demonstrates significant improvements in autonomous driving scene understanding.

Enables robust multimodal reasoning in complex environments.

Abstract

Vision-language models enable the understanding and reasoning of complex traffic scenarios through multi-source information fusion, establishing it as a core technology for autonomous driving. However, existing vision-language models are constrained by the image understanding paradigm in 2D plane, which restricts their capability to perceive 3D spatial information and perform deep semantic fusion, resulting in suboptimal performance in complex autonomous driving environments. This study proposes MMDrive, an multimodal vision-language model framework that extends traditional image understanding to a generalized 3D scene understanding framework. MMDrive incorporates three complementary modalities, including occupancy maps, LiDAR point clouds, and textual scene descriptions. To this end, it introduces two novel components for adaptive cross-modal fusion and key information extraction. Specifically, the Text-oriented Multimodal Modulator dynamically weights the contributions of each modality based on the semantic cues in the question, guiding context-aware feature integration. The Cross-Modal Abstractor employs learnable abstract tokens to generate compact, cross-modal summaries that highlight key regions and essential semantics. Comprehensive evaluations on the DriveLM and NuScenes-QA benchmarks demonstrate that MMDrive achieves significant performance gains over existing vision-language models for autonomous driving, with a BLEU-4 score of 54.56 and METEOR of 41.78 on DriveLM, and an accuracy score of 62.7% on NuScenes-QA. MMDrive effectively breaks the traditional image-only understanding barrier, enabling robust multimodal reasoning in complex driving environments and providing a new foundation for interpretable autonomous driving scene understanding.