MonoMVSNet: Monocular Priors Guided Multi-View Stereo Network

TL;DR

MonoMVSNet leverages monocular priors to enhance multi-view stereo depth estimation, especially in challenging regions, achieving state-of-the-art results on standard datasets by integrating monocular features and depths into the multi-view framework.

Contribution

The paper introduces MonoMVSNet, a novel MVS network that incorporates monocular priors through attention and dynamic depth updates, improving robustness and accuracy in difficult regions.

Findings

Achieves state-of-the-art performance on DTU and Tanks-and-Temples datasets.

Effectively handles textureless and reflective regions.

First to integrate monocular priors into multi-view stereo in this manner.

Abstract

Learning-based Multi-View Stereo (MVS) methods aim to predict depth maps for a sequence of calibrated images to recover dense point clouds. However, existing MVS methods often struggle with challenging regions, such as textureless regions and reflective surfaces, where feature matching fails. In contrast, monocular depth estimation inherently does not require feature matching, allowing it to achieve robust relative depth estimation in these regions. To bridge this gap, we propose MonoMVSNet, a novel monocular feature and depth guided MVS network that integrates powerful priors from a monocular foundation model into multi-view geometry. Firstly, the monocular feature of the reference view is integrated into source view features by the attention mechanism with a newly designed cross-view position encoding. Then, the monocular depth of the reference view is aligned to dynamically update the depth candidates for edge regions during the sampling procedure. Finally, a relative consistency loss is further designed based on the monocular depth to supervise the depth prediction. Extensive experiments demonstrate that MonoMVSNet achieves state-of-the-art performance on the DTU and Tanks-and-Temples datasets, ranking first on the Tanks-and-Temples Intermediate and Advanced benchmarks. The source code is available at https://github.com/JianfeiJ/MonoMVSNet.