Vision-Language Cross-Attention for Real-Time Autonomous Driving

TL;DR

This paper introduces XYZ-Drive, a vision-language model for autonomous driving that fuses image, map, and waypoint data using cross-attention, achieving high success rates and safety improvements in real-time navigation.

Contribution

The paper presents a novel single-model approach with goal-centered cross-attention for integrating vision, map, and waypoint data in autonomous driving, improving efficiency and performance.

Findings

XYZ-Drive achieves 95% success rate on MD-NEX benchmark.

Removing any modality reduces success, confirming their importance.

Query-based fusion outperforms simple concatenation in performance.

Abstract

Autonomous cars need geometric accuracy and semantic understanding to navigate complex environments, yet most stacks handle them separately. We present XYZ-Drive, a single vision-language model that reads a front-camera frame, a 25m $\times$ 25m overhead map, and the next waypoint, then outputs steering and speed. A lightweight goal-centered cross-attention layer lets waypoint tokens highlight relevant image and map patches, supporting both action and textual explanations, before the fused tokens enter a partially fine-tuned LLaMA-3.2 11B model. On the MD-NEX Outdoor-Driving benchmark XYZ-Drive attains 95% success and 0.80 Success weighted by Path Length (SPL), surpassing PhysNav-DG by 15%. and halving collisions, all while significantly improving efficiency by using only a single branch. Sixteen ablations explain the gains. Removing any modality (vision, waypoint, map) drops success by up to 11%, confirming their complementary roles and rich connections. Replacing goal-centered attention with simple concatenation cuts 3% in performance, showing query-based fusion injects map knowledge more effectively. Keeping the transformer frozen loses 5%, showing the importance of fine-tuning when applying VLMs for specific tasks such as autonomous driving. Coarsening map resolution from 10 cm to 40 cm blurs lane edges and raises crash rate. Overall, these results demonstrate that early, token-level fusion of intent and map layout enables accurate, transparent, real-time driving.