VLM-E2E: Enhancing End-to-End Autonomous Driving with Multimodal Driver Attention Fusion

TL;DR

VLM-E2E introduces a multimodal framework that enhances end-to-end autonomous driving by integrating vision-language models to improve semantic understanding and modality fusion, leading to better perception and decision-making.

Contribution

It is the first to incorporate textual attentional cues into BEV features for autonomous driving, addressing modality imbalance with a learnable fusion strategy.

Findings

Significant improvements in perception accuracy on nuScenes dataset.

Enhanced prediction and planning capabilities over baseline models.

Better alignment with human-like driving behavior.

Abstract

Human drivers adeptly navigate complex scenarios by utilizing rich attentional semantics, but the current autonomous systems struggle to replicate this ability, as they often lose critical semantic information when converting 2D observations into 3D space. In this sense, it hinders their effective deployment in dynamic and complex environments. Leveraging the superior scene understanding and reasoning abilities of Vision-Language Models (VLMs), we propose VLM-E2E, a novel framework that uses the VLMs to enhance training by providing attentional cues. Our method integrates textual representations into Bird's-Eye-View (BEV) features for semantic supervision, which enables the model to learn richer feature representations that explicitly capture the driver's attentional semantics. By focusing on attentional semantics, VLM-E2E better aligns with human-like driving behavior, which is critical for navigating dynamic and complex environments. Furthermore, we introduce a BEV-Text learnable weighted fusion strategy to address the issue of modality importance imbalance in fusing multimodal information. This approach dynamically balances the contributions of BEV and text features, ensuring that the complementary information from visual and textual modalities is effectively utilized. By explicitly addressing the imbalance in multimodal fusion, our method facilitates a more holistic and robust representation of driving environments. We evaluate VLM-E2E on the nuScenes dataset and achieve significant improvements in perception, prediction, and planning over the baseline end-to-end model, showcasing the effectiveness of our attention-enhanced BEV representation in enabling more accurate and reliable autonomous driving tasks.