StereoVGGT: A Training-Free Visual Geometry Transformer for Stereo Vision

TL;DR

StereoVGGT is a training-free, geometry-aware backbone for stereo vision that leverages a pretrained visual geometry transformer, significantly improving stereo matching performance on benchmarks.

Contribution

It introduces StereoVGGT, a novel training-free feature adjustment pipeline that enhances a pretrained visual geometry transformer for stereo vision tasks.

Findings

StereoVGGT achieved 1st rank on the KITTI benchmark.

The method effectively mitigates geometric degradation during feature extraction.

It demonstrates the benefit of using pretrained 3D priors for stereo vision.

Abstract

Driven by the advancement of 3D devices, stereo vision tasks including stereo matching and stereo conversion have emerged as a critical research frontier. Contemporary stereo vision backbones typically rely on either monocular depth estimation (MDE) models or visual foundation models (VFMs). Crucially, these models are predominantly pretrained without explicit supervision of camera poses. Given that such geometric knowledge is indispensable for stereo vision, the absence of explicit spatial constraints constitutes a significant performance bottleneck for existing architectures. Recognizing that the Visual Geometry Grounded Transformer (VGGT) operates as a foundation model pretrained on extensive 3D priors, including camera poses, we investigate its potential as a robust backbone for stereo vision tasks. Nevertheless, empirical results indicate that its direct application to stereo vision yields suboptimal performance. We observe that VGGT suffers from a more significant degradation of geometric details during feature extraction. Such characteristics conflict with the requirements of binocular stereo vision, thereby constraining its efficacy for relative tasks. To bridge this gap, we propose StereoVGGT, a feature backbone specifically tailored for stereo vision. By leveraging the frozen VGGT and introducing a training-free feature adjustment pipeline, we mitigate geometric degradation and harness the latent camera calibration knowledge embedded within the model. StereoVGGT-based stereo matching network achieved the $1^{st}$ rank among all published methods on the KITTI benchmark, validating that StereoVGGT serves as a highly effective backbone for stereo vision.