LightEMMA: Lightweight End-to-End Multimodal Model for Autonomous Driving

TL;DR

LightEMMA introduces a flexible, lightweight multimodal framework for autonomous driving that facilitates model updates, validation, and comparison, highlighting the current strengths and limitations of vision-language models in real-world scenarios.

Contribution

We present LightEMMA, a unified, adaptable VLM-based autonomous driving framework enabling easy integration, evaluation, and comparison of various models without custom modifications.

Findings

VLMs show strong scenario interpretation capabilities.

Increased complexity does not guarantee better performance.

Practical performance of VLMs in driving remains a challenge.

Abstract

Vision-Language Models (VLMs) have demonstrated significant potential for end-to-end autonomous driving. However, the field still lacks a practical platform that enables dynamic model updates, rapid validation, fair comparison, and intuitive performance assessment. To that end, we introduce LightEMMA, a Lightweight End-to-End Multimodal Model for Autonomous driving. LightEMMA provides a unified, VLM-based autonomous driving framework without ad hoc customizations, enabling easy integration with evolving state-of-the-art commercial and open-source models. We construct twelve autonomous driving agents using various VLMs and evaluate their performance on the challenging nuScenes prediction task, comprehensively assessing computational metrics and providing critical insights. Illustrative examples show that, although VLMs exhibit strong scenario interpretation capabilities, their practical performance in autonomous driving tasks remains a concern. Additionally, increased model complexity and extended reasoning do not necessarily lead to better performance, emphasizing the need for further improvements and task-specific designs. The code is available at https://github.com/michigan-traffic-lab/LightEMMA.