VLP: Vision Language Planning for Autonomous Driving

TL;DR

VLP introduces a vision-language planning framework that leverages language models to improve autonomous driving by enhancing scene understanding, reasoning, and generalization, achieving state-of-the-art results on NuScenes.

Contribution

The paper presents VLP, a novel framework integrating language models into autonomous driving to address reasoning and generalization challenges.

Findings

Achieves 35.9% reduction in average L2 error

Achieves 60.5% reduction in collision rates

Improves performance in long-tail scenarios

Abstract

Autonomous driving is a complex and challenging task that aims at safe motion planning through scene understanding and reasoning. While vision-only autonomous driving methods have recently achieved notable performance, through enhanced scene understanding, several key issues, including lack of reasoning, low generalization performance and long-tail scenarios, still need to be addressed. In this paper, we present VLP, a novel Vision-Language-Planning framework that exploits language models to bridge the gap between linguistic understanding and autonomous driving. VLP enhances autonomous driving systems by strengthening both the source memory foundation and the self-driving car's contextual understanding. VLP achieves state-of-the-art end-to-end planning performance on the challenging NuScenes dataset by achieving 35.9\% and 60.5\% reduction in terms of average L2 error and collision rates, respectively, compared to the previous best method. Moreover, VLP shows improved performance in challenging long-tail scenarios and strong generalization capabilities when faced with new urban environments.