Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving

TL;DR

This paper introduces EM-VLM4AD, a lightweight, multi-frame vision-language model designed for real-time autonomous driving question answering, significantly reducing memory and computation while improving accuracy over existing models.

Contribution

The paper presents EM-VLM4AD, a novel efficient multi-frame vision-language model that outperforms existing approaches in memory usage, speed, and accuracy for autonomous driving tasks.

Findings

Requires 10x less memory and FLOPs than baselines.

Achieves higher CIDEr and ROUGE-L scores.

Effectively extracts relevant traffic scene information.

Abstract

Vision-Language Models (VLMs) and Multi-Modal Language models (MMLMs) have become prominent in autonomous driving research, as these models can provide interpretable textual reasoning and responses for end-to-end autonomous driving safety tasks using traffic scene images and other data modalities. However, current approaches to these systems use expensive large language model (LLM) backbones and image encoders, making such systems unsuitable for real-time autonomous driving systems where tight memory constraints exist and fast inference time is necessary. To address these previous issues, we develop EM-VLM4AD, an efficient, lightweight, multi-frame vision language model which performs Visual Question Answering for autonomous driving. In comparison to previous approaches, EM-VLM4AD requires at least 10 times less memory and floating point operations, while also achieving higher CIDEr and ROUGE-L scores than the existing baseline on the DriveLM dataset. EM-VLM4AD also exhibits the ability to extract relevant information from traffic views related to prompts and can answer questions for various autonomous driving subtasks. We release our code to train and evaluate our model at https://github.com/akshaygopalkr/EM-VLM4AD.