DVGT-2: Vision-Geometry-Action Model for Autonomous Driving at Scale

TL;DR

DVGT-2 introduces an online, efficient dense 3D geometry and trajectory planning model for autonomous driving, outperforming previous methods in speed and versatility across multiple datasets.

Contribution

It proposes a streaming DVGT-2 model that processes inputs in real-time, jointly reconstructs geometry and plans trajectories without fine-tuning for different camera setups.

Findings

DVGT-2 achieves superior geometry reconstruction performance.

The model works across diverse datasets without fine-tuning.

It operates efficiently with a sliding-window streaming strategy.

Abstract

End-to-end autonomous driving has evolved from the conventional paradigm based on sparse perception into vision-language-action (VLA) models, which focus on learning language descriptions as an auxiliary task to facilitate planning. In this paper, we propose an alternative Vision-Geometry-Action (VGA) paradigm that advocates dense 3D geometry as the critical cue for autonomous driving. As vehicles operate in a 3D world, we think dense 3D geometry provides the most comprehensive information for decision-making. However, most existing geometry reconstruction methods (e.g., DVGT) rely on computationally expensive batch processing of multi-frame inputs and cannot be applied to online planning. To address this, we introduce a streaming Driving Visual Geometry Transformer (DVGT-2), which processes inputs in an online manner and jointly outputs dense geometry and trajectory planning for the current frame. We employ temporal causal attention and cache historical features to support on-the-fly inference. To further enhance efficiency, we propose a sliding-window streaming strategy and use historical caches within a certain interval to avoid repetitive computations. Despite the faster speed, DVGT-2 achieves superior geometry reconstruction performance on various datasets. The same trained DVGT-2 can be directly applied to planning across diverse camera configurations without fine-tuning, including closed-loop NAVSIM and open-loop nuScenes benchmarks.