V2X-UniPool: Unifying Multimodal Perception and Knowledge Reasoning for Autonomous Driving

TL;DR

V2X-UniPool is a novel framework that integrates multimodal V2X perception with language-based reasoning, enhancing autonomous driving safety and efficiency by unifying sensor data and knowledge reasoning.

Contribution

It is the first to unify V2X perception with language reasoning, transforming multimodal data into structured knowledge for improved decision-making in autonomous driving.

Findings

Achieves state-of-the-art planning accuracy and safety.

Reduces communication cost by over 80%.

Maintains low overhead compared to other methods.

Abstract

Autonomous driving (AD) has achieved significant progress, yet single-vehicle perception remains constrained by sensing range and occlusions. Vehicle-to-Everything (V2X) communication addresses these limits by enabling collaboration across vehicles and infrastructure, but it also faces heterogeneity, synchronization, and latency constraints. Language models offer strong knowledge-driven reasoning and decision-making capabilities, but they are not inherently designed to process raw sensor streams and are prone to hallucination. We propose V2X-UniPool, the first framework that unifies V2X perception with language-based reasoning for knowledge-driven AD. It transforms multimodal V2X data into structured, language-based knowledge, organizes it in a time-indexed knowledge pool for temporally consistent reasoning, and employs Retrieval-Augmented Generation (RAG) to ground decisions in real-time context. Experiments on the real-world DAIR-V2X dataset show that V2X-UniPool achieves state-of-the-art planning accuracy and safety while reducing communication cost by more than 80\%, achieving the lowest overhead among evaluated methods. These results highlight the promise of bridging V2X perception and language reasoning to advance scalable and trustworthy driving. Our code is available at: https://github.com/Xuewen2025/V2X-UniPool