Multi-view Surface Reconstruction Using Normal and Reflectance Cues

TL;DR

This paper presents a versatile multi-view surface reconstruction framework that integrates normal and reflectance cues, achieving state-of-the-art results in fine detail preservation and challenging visibility conditions using both traditional and neural rendering methods.

Contribution

It introduces a joint re-parametrization of reflectance and normals as radiance vectors, enabling seamless integration into existing surface reconstruction pipelines, with improved robustness and speed.

Findings

State-of-the-art performance on MVPS benchmarks

Superior fine-detail reconstruction in complex materials

Enhanced robustness and efficiency of the algorithm

Abstract

Achieving high-fidelity 3D surface reconstruction while preserving fine details remains challenging, especially in the presence of materials with complex reflectance properties and without a dense-view setup. In this paper, we introduce a versatile framework that incorporates multi-view normal and optionally reflectance maps into radiance-based surface reconstruction. Our approach employs a pixel-wise joint re-parametrization of reflectance and surface normals, representing them as a vector of radiances under simulated, varying illumination. This formulation enables seamless incorporation into standard surface reconstruction pipelines, such as traditional multi-view stereo (MVS) frameworks or modern neural volume rendering (NVR) ones. Combined with the latter, our approach achieves state-of-the-art performance on multi-view photometric stereo (MVPS) benchmark datasets, including DiLiGenT-MV, LUCES-MV and Skoltech3D. In particular, our method excels in reconstructing fine-grained details and handling challenging visibility conditions. The present paper is an extended version of the earlier conference paper by Brument et al. (in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024), featuring an accelerated and more robust algorithm as well as a broader empirical evaluation. The code and data relative to this article is available at https://github.com/RobinBruneau/RNb-NeuS2.