Visual Implicit Geometry Transformer for Autonomous Driving

TL;DR

ViGT is a scalable, calibration-free transformer model that estimates continuous 3D occupancy fields from surround-view cameras for autonomous driving, demonstrating state-of-the-art results across multiple large-scale datasets.

Contribution

Introduces ViGT, a novel self-supervised, calibration-free geometric model for autonomous driving that generalizes across diverse sensor configurations and datasets.

Findings

Achieves state-of-the-art pointmap estimation performance.

Performs comparably to supervised methods on Occ3D-nuScenes.

Demonstrates strong generalization across five large-scale datasets.

Abstract

We introduce the Visual Implicit Geometry Transformer (ViGT), an autonomous driving geometric model that estimates continuous 3D occupancy fields from surround-view camera rigs. ViGT represents a step towards foundational geometric models for autonomous driving, prioritizing scalability, architectural simplicity, and generalization across diverse sensor configurations. Our approach achieves this through a calibration-free architecture, enabling a single model to adapt to different sensor setups. Unlike general-purpose geometric foundational models that focus on pixel-aligned predictions, ViGT estimates a continuous 3D occupancy field in a birds-eye-view (BEV) addressing domain-specific requirements. ViGT naturally infers geometry from multiple camera views into a single metric coordinate frame, providing a common representation for multiple geometric tasks. Unlike most existing occupancy models, we adopt a self-supervised training procedure that leverages synchronized image-LiDAR pairs, eliminating the need for costly manual annotations. We validate the scalability and generalizability of our approach by training our model on a mixture of five large-scale autonomous driving datasets (NuScenes, Waymo, NuPlan, ONCE, and Argoverse) and achieving state-of-the-art performance on the pointmap estimation task, with the best average rank across all evaluated baselines. We further evaluate ViGT on the Occ3D-nuScenes benchmark, where ViGT achieves comparable performance with supervised methods. The source code is publicly available at \href{https://github.com/whesense/ViGT}{https://github.com/whesense/ViGT}.