Unlocking the Power of Critical Factors for 3D Visual Geometry Estimation

TL;DR

This paper systematically investigates critical factors affecting 3D visual geometry estimation, revealing data quality, supervision strategies, and high-resolution inputs significantly influence model performance.

Contribution

The authors introduce CARVE, a resolution-enhanced model that integrates new consistency loss and architectural improvements for better 3D geometry estimation.

Findings

Scaling data diversity improves performance.

Certain loss mechanisms may hinder accuracy.

Joint supervision enhances results, local region alignment can degrade.

Abstract

Feed-forward visual geometry estimation has recently made rapid progress. However, an important gap remains: multi-frame models usually produce better cross-frame consistency, yet they often underperform strong per-frame methods on single-frame accuracy. This observation motivates our systematic investigation into the critical factors driving model performance through rigorous ablation studies, which reveals several key insights: 1) Scaling up data diversity and quality unlocks further performance gains even in state-of-the-art visual geometry estimation methods; 2) Commonly adopted confidence-aware loss and gradient-based loss mechanisms may unintentionally hinder performance; 3) Joint supervision through both per-sequence and per-frame alignment improves results, while local region alignment surprisingly degrades performance. Furthermore, we introduce two enhancements to integrate the advantages of optimization-based methods and high-resolution inputs: a consistency loss function that enforces alignment between depth maps, camera parameters, and point maps, and an efficient architectural design that leverages high-resolution information. We integrate these designs into CARVE, a resolution-enhanced model for feed-forward visual geometry estimation. Experiments on point cloud reconstruction, video depth estimation, and camera pose/intrinsic estimation show that CARVE achieves strong and robust performance across diverse benchmarks.