From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Driving

TL;DR

This paper investigates how vision-language models (VLM) and vision-only backbones can be combined for end-to-end driving, revealing their complementary behaviors and proposing a hybrid system that improves decision accuracy and efficiency.

Contribution

It introduces a systematic analysis of VLM and vision-only backbones in driving, and proposes HybridDriveVLA and DualDriveVLA systems that leverage their complementarity for better performance.

Findings

VLM introduces additional subspaces in the feature space.

VLM tends to be more aggressive in long-tail scenarios.

HybridDriveVLA achieves 92.10 PDMS, and DualDriveVLA improves throughput by 3.2x.

Abstract

Vision-Language-Action (VLA) driving augments end-to-end (E2E) planning with language-enabled backbones, yet it remains unclear what changes beyond the usual accuracy--cost trade-off. We revisit this question with 3--RQ analysis in RecogDrive by instantiating the system with a full VLM and vision-only backbones, all under an identical diffusion Transformer planner. RQ1: At the backbone level, the VLM can introduce additional subspaces upon the vision-only backbones. RQ2: This unique subspace leads to a different behavioral in some long-tail scenario: the VLM tends to be more aggressive whereas ViT is more conservative, and each decisively wins on about 2--3% of test scenarios; With an oracle that selects, per scenario, the better trajectory between the VLM and ViT branches, we obtain an upper bound of 93.58 PDMS. RQ3: To fully harness this observation, we propose HybridDriveVLA, which runs both ViT and VLM branches and selects between their endpoint trajectories using a learned scorer, improving PDMS to 92.10. Finally, DualDriveVLA implements a practical fast--slow policy: it runs ViT by default and invokes the VLM only when the scorer's confidence falls below a threshold; calling the VLM on 15% of scenarios achieves 91.00 PDMS while improving throughput by 3.2x. Code will be released.