Neural apparent BRDF fields for multiview photometric stereo

TL;DR

This paper introduces a neural approach extending NeRFs to multiview photometric stereo, modeling surface normals and reflectance to improve 3D shape and appearance reconstruction from multiple views and lighting conditions.

Contribution

It presents a novel neural representation that combines surface normal prediction with a learned BRDF and shadow modeling, enabling better extrapolation and reflectance modeling in multiview photometric stereo.

Findings

Achieves competitive results on a multiview photometric stereo benchmark.

Effectively models local surface reflectance and normal directions.

Demonstrates extrapolation capabilities beyond observed lighting conditions.

Abstract

We propose to tackle the multiview photometric stereo problem using an extension of Neural Radiance Fields (NeRFs), conditioned on light source direction. The geometric part of our neural representation predicts surface normal direction, allowing us to reason about local surface reflectance. The appearance part of our neural representation is decomposed into a neural bidirectional reflectance function (BRDF), learnt as part of the fitting process, and a shadow prediction network (conditioned on light source direction) allowing us to model the apparent BRDF. This balance of learnt components with inductive biases based on physical image formation models allows us to extrapolate far from the light source and viewer directions observed during training. We demonstrate our approach on a multiview photometric stereo benchmark and show that competitive performance can be obtained with the neural density representation of a NeRF.