NeIF: Representing General Reflectance as Neural Intrinsics Fields for Uncalibrated Photometric Stereo

TL;DR

This paper introduces NeIF, a novel neural intrinsic fields approach for uncalibrated photometric stereo that jointly models reflectance, light, specular, and shadow to improve 3D shape recovery without supervision.

Contribution

It proposes representing reflectance as four neural intrinsics fields, enabling unsupervised joint optimization that surpasses existing methods in accuracy and robustness.

Findings

Outperforms state-of-the-art UPS methods on multiple datasets.

Effectively handles challenging lighting and reflectance conditions.

Achieves superior shape and reflectance estimation accuracy.

Abstract

Uncalibrated photometric stereo (UPS) is challenging due to the inherent ambiguity brought by unknown light. Existing solutions alleviate the ambiguity by either explicitly associating reflectance to light conditions or resolving light conditions in a supervised manner. This paper establishes an implicit relation between light clues and light estimation and solves UPS in an unsupervised manner. The key idea is to represent the reflectance as four neural intrinsics fields, i.e., position, light, specular, and shadow, based on which the neural light field is implicitly associated with light clues of specular reflectance and cast shadow. The unsupervised, joint optimization of neural intrinsics fields can be free from training data bias as well as accumulating error, and fully exploits all observed pixel values for UPS. Our method achieves a superior performance advantage over state-of-the-art UPS methods on public and self-collected datasets, under regular and challenging setups. The code will be released soon.