DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiences

TL;DR

DriVLMe is a novel video-language-model-based agent designed to improve human-vehicle communication and navigation in autonomous driving, trained on embodied and social experiences, revealing both capabilities and current limitations.

Contribution

The paper introduces DriVLMe, integrating embodied and social experiences into a unified model for autonomous driving communication and navigation tasks.

Findings

Competitive performance in benchmarks and human studies.

Identified limitations such as inference time and visual understanding.

Challenges with multi-turn interactions and unexpected situations.

Abstract

Recent advancements in foundation models (FMs) have unlocked new prospects in autonomous driving, yet the experimental settings of these studies are preliminary, over-simplified, and fail to capture the complexity of real-world driving scenarios in human environments. It remains under-explored whether FM agents can handle long-horizon navigation tasks with free-from dialogue and deal with unexpected situations caused by environmental dynamics or task changes. To explore the capabilities and boundaries of FMs faced with the challenges above, we introduce DriVLMe, a video-language-model-based agent to facilitate natural and effective communication between humans and autonomous vehicles that perceive the environment and navigate. We develop DriVLMe from both embodied experiences in a simulated environment and social experiences from real human dialogue. While DriVLMe demonstrates competitive performance in both open-loop benchmarks and closed-loop human studies, we reveal several limitations and challenges, including unacceptable inference time, imbalanced training data, limited visual understanding, challenges with multi-turn interactions, simplified language generation from robotic experiences, and difficulties in handling on-the-fly unexpected situations like environmental dynamics and task changes.