Learning-based Multi-View Stereo: A Survey

TL;DR

This survey reviews recent learning-based multi-view stereo methods for 3D reconstruction, categorizing approaches, analyzing their performance on benchmarks, and discussing future research directions in the field.

Contribution

It provides a comprehensive overview of learning-based MVS methods, emphasizing depth map-based approaches and summarizing their performance and future prospects.

Findings

Learning-based MVS methods outperform traditional approaches.

Depth map-based methods are the most prominent due to their efficiency.

Benchmark results highlight the progress and challenges in the field.

Abstract

3D reconstruction aims to recover the dense 3D structure of a scene. It plays an essential role in various applications such as Augmented/Virtual Reality (AR/VR), autonomous driving and robotics. Leveraging multiple views of a scene captured from different viewpoints, Multi-View Stereo (MVS) algorithms synthesize a comprehensive 3D representation, enabling precise reconstruction in complex environments. Due to its efficiency and effectiveness, MVS has become a pivotal method for image-based 3D reconstruction. Recently, with the success of deep learning, many learning-based MVS methods have been proposed, achieving impressive performance against traditional methods. We categorize these learning-based methods as: depth map-based, voxel-based, NeRF-based, 3D Gaussian Splatting-based, and large feed-forward methods. Among these, we focus significantly on depth map-based methods, which are the main family of MVS due to their conciseness, flexibility and scalability. In this survey, we provide a comprehensive review of the literature at the time of this writing. We investigate these learning-based methods, summarize their performances on popular benchmarks, and discuss promising future research directions in this area.