A Confidence-based Iterative Solver of Depths and Surface Normals for Deep Multi-view Stereo

TL;DR

This paper presents a novel confidence-based iterative solver integrated into a deep multi-view stereo system that jointly predicts depths and surface normals, improving multi-view depth estimation accuracy.

Contribution

The paper introduces a differentiable, confidence-guided iterative solver for depth and normal estimation within a deep MVS framework, enabling end-to-end training and improved results.

Findings

Achieves state-of-the-art performance on ScanNet and RGB-D Scenes V2.

Effectively improves depth quality over existing methods.

Demonstrates the solver's versatility as both a post-processing tool and an integrated component.

Abstract

In this paper, we introduce a deep multi-view stereo (MVS) system that jointly predicts depths, surface normals and per-view confidence maps. The key to our approach is a novel solver that iteratively solves for per-view depth map and normal map by optimizing an energy potential based on the locally planar assumption. Specifically, the algorithm updates depth map by propagating from neighboring pixels with slanted planes, and updates normal map with local probabilistic plane fitting. Both two steps are monitored by a customized confidence map. This solver is not only effective as a post-processing tool for plane-based depth refinement and completion, but also differentiable such that it can be efficiently integrated into deep learning pipelines. Our multi-view stereo system employs multiple optimization steps of the solver over the initial prediction of depths and surface normals. The whole system can be trained end-to-end, decoupling the challenging problem of matching pixels within poorly textured regions from the cost-volume based neural network. Experimental results on ScanNet and RGB-D Scenes V2 demonstrate state-of-the-art performance of the proposed deep MVS system on multi-view depth estimation, with our proposed solver consistently improving the depth quality over both conventional and deep learning based MVS pipelines. Code is available at https://github.com/thuzhaowang/idn-solver.