DVGT: Driving Visual Geometry Transformer

TL;DR

DVGT is a novel transformer-based model that reconstructs dense 3D scene geometry directly from multi-view visual inputs in autonomous driving, without relying on explicit camera calibration or external sensors.

Contribution

It introduces a flexible, camera-agnostic transformer architecture for dense 3D reconstruction from multi-view images in driving scenarios, outperforming existing methods.

Findings

Outperforms existing models on multiple driving datasets.

Does not require explicit camera parameters or external sensors.

Successfully reconstructs metric-scaled 3D geometry from image sequences.

Abstract

Perceiving and reconstructing 3D scene geometry from visual inputs is crucial for autonomous driving. However, there still lacks a driving-targeted dense geometry perception model that can adapt to different scenarios and camera configurations. To bridge this gap, we propose a Driving Visual Geometry Transformer (DVGT), which reconstructs a global dense 3D point map from a sequence of unposed multi-view visual inputs. We first extract visual features for each image using a DINO backbone, and employ alternating intra-view local attention, cross-view spatial attention, and cross-frame temporal attention to infer geometric relations across images. We then use multiple heads to decode a global point map in the ego coordinate of the first frame and the ego poses for each frame. Unlike conventional methods that rely on precise camera parameters, DVGT is free of explicit 3D geometric priors, enabling flexible processing of arbitrary camera configurations. DVGT directly predicts metric-scaled geometry from image sequences, eliminating the need for post-alignment with external sensors. Trained on a large mixture of driving datasets including nuScenes, OpenScene, Waymo, KITTI, and DDAD, DVGT significantly outperforms existing models on various scenarios. Code is available at https://github.com/wzzheng/DVGT.