nLMVS-Net: Deep Non-Lambertian Multi-View Stereo

TL;DR

nLMVS-Net is a deep learning approach for multi-view stereo that jointly estimates depth, surface normals, and reflectance of complex non-Lambertian surfaces under natural illumination, improving shape and reflectance recovery.

Contribution

It introduces an end-to-end learnable network that integrates radiometric cues and surface normals for robust multi-view stereo reconstruction of non-Lambertian surfaces.

Findings

Accurately recovers shape and reflectance of complex objects.

Performs well on synthetic and real-world datasets.

Robustly handles natural lighting conditions.

Abstract

We introduce a novel multi-view stereo (MVS) method that can simultaneously recover not just per-pixel depth but also surface normals, together with the reflectance of textureless, complex non-Lambertian surfaces captured under known but natural illumination. Our key idea is to formulate MVS as an end-to-end learnable network, which we refer to as nLMVS-Net, that seamlessly integrates radiometric cues to leverage surface normals as view-independent surface features for learned cost volume construction and filtering. It first estimates surface normals as pixel-wise probability densities for each view with a novel shape-from-shading network. These per-pixel surface normal densities and the input multi-view images are then input to a novel cost volume filtering network that learns to recover per-pixel depth and surface normal. The reflectance is also explicitly estimated by alternating with geometry reconstruction. Extensive quantitative evaluations on newly established synthetic and real-world datasets show that nLMVS-Net can robustly and accurately recover the shape and reflectance of complex objects in natural settings.