Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving

TL;DR

This paper introduces a structured dataset and a compact vision-language model for autonomous driving, significantly improving decision accuracy and inference speed by leveraging structured scene descriptions.

Contribution

The paper presents NuScenes-S, a structured dataset, and FastDrive, a lightweight VLM, enhancing efficiency and decision-making in autonomous driving applications.

Findings

FastDrive achieves 20% higher decision accuracy.

Over 10x inference speedup compared to larger models.

Scene annotations significantly impact decision-making performance.

Abstract

Vision-Language Models (VLMs) offer a promising approach to end-to-end autonomous driving due to their human-like reasoning capabilities. However, troublesome gaps remains between current VLMs and real-world autonomous driving applications. One major limitation is that existing datasets with loosely formatted language descriptions are not machine-friendly and may introduce redundancy. Additionally, high computational cost and massive scale of VLMs hinder the inference speed and real-world deployment. To bridge the gap, this paper introduces a structured and concise benchmark dataset, NuScenes-S, which is derived from the NuScenes dataset and contains machine-friendly structured representations. Moreover, we present FastDrive, a compact VLM baseline with 0.9B parameters. In contrast to existing VLMs with over 7B parameters and unstructured language processing(e.g., LLaVA-1.5), FastDrive understands structured and concise descriptions and generates machine-friendly driving decisions with high efficiency. Extensive experiments show that FastDrive achieves competitive performance on structured dataset, with approximately 20% accuracy improvement on decision-making tasks, while surpassing massive parameter baseline in inference speed with over 10x speedup. Additionally, ablation studies further focus on the impact of scene annotations (e.g., weather, time of day) on decision-making tasks, demonstrating their importance on decision-making tasks in autonomous driving.