PX-NET: Simple and Efficient Pixel-Wise Training of Photometric Stereo Networks

TL;DR

This paper introduces PX-NET, a pixel-wise training method for photometric stereo that simplifies data generation by approximating global illumination effects, leading to state-of-the-art 3D surface normal predictions.

Contribution

The paper presents a novel pixel-wise training procedure that replaces complex global illumination rendering with independent per-pixel data, enabling efficient and accurate normal prediction.

Findings

Achieves state-of-the-art performance on synthetic datasets.

Performs well on real Diligent dataset with dense and sparse lighting.

Speeds up training data creation by approximating global effects.

Abstract

Retrieving accurate 3D reconstructions of objects from the way they reflect light is a very challenging task in computer vision. Despite more than four decades since the definition of the Photometric Stereo problem, most of the literature has had limited success when global illumination effects such as cast shadows, self-reflections and ambient light come into play, especially for specular surfaces. Recent approaches have leveraged the power of deep learning in conjunction with computer graphics in order to cope with the need of a vast number of training data in order to invert the image irradiance equation and retrieve the geometry of the object. However, rendering global illumination effects is a slow process which can limit the amount of training data that can be generated. In this work we propose a novel pixel-wise training procedure for normal prediction by replacing the training data (observation maps) of globally rendered images with independent per-pixel generated data. We show that global physical effects can be approximated on the observation map domain and this simplifies and speeds up the data creation procedure. Our network, PX-NET, achieves the state-of-the-art performance compared to other pixelwise methods on synthetic datasets, as well as the Diligent real dataset on both dense and sparse light settings.