Uncertainty-Aware Deep Multi-View Photometric Stereo

TL;DR

This paper introduces an uncertainty-aware method combining deep multi-view photometric stereo and multi-view stereo for high-quality, dense surface reconstruction, outperforming existing approaches with lower memory use.

Contribution

It presents a novel approach that integrates uncertainty modeling in deep PS and MVS networks to improve dense surface reconstruction accuracy.

Findings

Outperforms existing methods on DiLiGenT-MV benchmark

Achieves high-quality shape recovery with lower memory footprint

Effectively combines PS and MVS strengths using uncertainty estimates

Abstract

This paper presents a simple and effective solution to the longstanding classical multi-view photometric stereo (MVPS) problem. It is well-known that photometric stereo (PS) is excellent at recovering high-frequency surface details, whereas multi-view stereo (MVS) can help remove the low-frequency distortion due to PS and retain the global geometry of the shape. This paper proposes an approach that can effectively utilize such complementary strengths of PS and MVS. Our key idea is to combine them suitably while considering the per-pixel uncertainty of their estimates. To this end, we estimate per-pixel surface normals and depth using an uncertainty-aware deep-PS network and deep-MVS network, respectively. Uncertainty modeling helps select reliable surface normal and depth estimates at each pixel which then act as a true representative of the dense surface geometry. At each pixel, our approach either selects or discards deep-PS and deep-MVS network prediction depending on the prediction uncertainty measure. For dense, detailed, and precise inference of the object's surface profile, we propose to learn the implicit neural shape representation via a multilayer perceptron (MLP). Our approach encourages the MLP to converge to a natural zero-level set surface using the confident prediction from deep-PS and deep-MVS networks, providing superior dense surface reconstruction. Extensive experiments on the DiLiGenT-MV benchmark dataset show that our method provides high-quality shape recovery with a much lower memory footprint while outperforming almost all of the existing approaches.