SOLVE: Synergy of Language-Vision and End-to-End Networks for Autonomous Driving

TL;DR

SOLVE integrates vision-language models with end-to-end autonomous driving systems using a shared visual encoder and a trajectory refinement paradigm, significantly improving trajectory prediction accuracy and system robustness.

Contribution

The paper introduces a novel framework combining VLMs and E2E models with a shared encoder and a trajectory refinement method for better autonomous driving performance.

Findings

Enhanced trajectory prediction accuracy on nuScenes dataset

Effective knowledge sharing between VLMs and E2E models

Improved real-time decision-making efficiency

Abstract

The integration of Vision-Language Models (VLMs) into autonomous driving systems has shown promise in addressing key challenges such as learning complexity, interpretability, and common-sense reasoning. However, existing approaches often struggle with efficient integration and realtime decision-making due to computational demands. In this paper, we introduce SOLVE, an innovative framework that synergizes VLMs with end-to-end (E2E) models to enhance autonomous vehicle planning. Our approach emphasizes knowledge sharing at the feature level through a shared visual encoder, enabling comprehensive interaction between VLM and E2E components. We propose a Trajectory Chain-of-Thought (T-CoT) paradigm, which progressively refines trajectory predictions, reducing uncertainty and improving accuracy. By employing a temporal decoupling strategy, SOLVE achieves efficient cooperation by aligning high-quality VLM outputs with E2E real-time performance. Evaluated on the nuScenes dataset, our method demonstrates significant improvements in trajectory prediction accuracy, paving the way for more robust and reliable autonomous driving systems.