3DVNet: Multi-View Depth Prediction and Volumetric Refinement

TL;DR

3DVNet introduces a multi-view stereo depth prediction method that combines volumetric 3D CNNs and iterative refinement to achieve superior accuracy in 3D scene reconstruction.

Contribution

The paper proposes a novel 3D CNN-based multi-view stereo approach that iteratively refines depth maps using scene-level priors and feature-augmented point clouds.

Findings

Outperforms state-of-the-art in depth prediction accuracy

Achieves superior 3D reconstruction quality

Generalizes well across different datasets

Abstract

We present 3DVNet, a novel multi-view stereo (MVS) depth-prediction method that combines the advantages of previous depth-based and volumetric MVS approaches. Our key idea is the use of a 3D scene-modeling network that iteratively updates a set of coarse depth predictions, resulting in highly accurate predictions which agree on the underlying scene geometry. Unlike existing depth-prediction techniques, our method uses a volumetric 3D convolutional neural network (CNN) that operates in world space on all depth maps jointly. The network can therefore learn meaningful scene-level priors. Furthermore, unlike existing volumetric MVS techniques, our 3D CNN operates on a feature-augmented point cloud, allowing for effective aggregation of multi-view information and flexible iterative refinement of depth maps. Experimental results show our method exceeds state-of-the-art accuracy in both depth prediction and 3D reconstruction metrics on the ScanNet dataset, as well as a selection of scenes from the TUM-RGBD and ICL-NUIM datasets. This shows that our method is both effective and generalizes to new settings.