DeepPS2: Revisiting Photometric Stereo Using Two Differently Illuminated Images

TL;DR

DeepPS2 introduces a self-supervised deep learning framework that accurately estimates surface normals, albedo, and lighting from just two images, addressing limitations of traditional photometric stereo methods.

Contribution

It presents the first self-supervised method for photometric stereo using only two images, jointly estimating surface normals, albedo, and lighting without ground truth data.

Findings

Effective surface normal and lighting estimation from two images.

Improved accuracy through joint relighting and reconstruction.

No need for ground truth normals or known lighting during training.

Abstract

Photometric stereo, a problem of recovering 3D surface normals using images of an object captured under different lightings, has been of great interest and importance in computer vision research. Despite the success of existing traditional and deep learning-based methods, it is still challenging due to: (i) the requirement of three or more differently illuminated images, (ii) the inability to model unknown general reflectance, and (iii) the requirement of accurate 3D ground truth surface normals and known lighting information for training. In this work, we attempt to address an under-explored problem of photometric stereo using just two differently illuminated images, referred to as the PS2 problem. It is an intermediate case between a single image-based reconstruction method like Shape from Shading (SfS) and the traditional Photometric Stereo (PS), which requires three or more images. We propose an inverse rendering-based deep learning framework, called DeepPS2, that jointly performs surface normal, albedo, lighting estimation, and image relighting in a completely self-supervised manner with no requirement of ground truth data. We demonstrate how image relighting in conjunction with image reconstruction enhances the lighting estimation in a self-supervised setting.