Review of Feed-forward 3D Reconstruction: From DUSt3R to VGGT

TL;DR

This paper systematically reviews feed-forward deep learning models for 3D reconstruction, highlighting their technical frameworks, advantages over traditional methods, and future challenges in scalability and dynamic scene handling.

Contribution

It provides a comprehensive analysis of feed-forward 3D reconstruction models like DUSt3R, contrasting them with traditional and earlier learning-based methods, and discusses future research directions.

Findings

Feed-forward models enable direct, single-pass 3D reconstruction from images.

Transformer-based correspondence modeling improves accuracy.

Challenges include scalability and dynamic scene handling.

Abstract

3D reconstruction, which aims to recover the dense three-dimensional structure of a scene, is a cornerstone technology for numerous applications, including augmented/virtual reality, autonomous driving, and robotics. While traditional pipelines like Structure from Motion (SfM) and Multi-View Stereo (MVS) achieve high precision through iterative optimization, they are limited by complex workflows, high computational cost, and poor robustness in challenging scenarios like texture-less regions. Recently, deep learning has catalyzed a paradigm shift in 3D reconstruction. A new family of models, exemplified by DUSt3R, has pioneered a feed-forward approach. These models employ a unified deep network to jointly infer camera poses and dense geometry directly from an Unconstrained set of images in a single forward pass. This survey provides a systematic review of this emerging domain. We begin by dissecting the technical framework of these feed-forward models, including their Transformer-based correspondence modeling, joint pose and geometry regression mechanisms, and strategies for scaling from two-view to multi-view scenarios. To highlight the disruptive nature of this new paradigm, we contrast it with both traditional pipelines and earlier learning-based methods like MVSNet. Furthermore, we provide an overview of relevant datasets and evaluation metrics. Finally, we discuss the technology's broad application prospects and identify key future challenges and opportunities, such as model accuracy and scalability, and handling dynamic scenes.