Differentiable Display Photometric Stereo

TL;DR

This paper introduces differentiable display photometric stereo (DDPS), a method that learns optimal display patterns for accurate surface normal reconstruction using monitors, with a differentiable framework and real-world training data.

Contribution

DDPS proposes a novel end-to-end differentiable framework that learns display patterns for photometric stereo, improving accuracy and robustness over heuristic patterns.

Findings

Enhanced normal reconstruction accuracy compared to heuristic patterns

Robustness to pattern initialization and calibration errors

Effective separation of diffuse and specular reflections using polarization

Abstract

Photometric stereo leverages variations in illumination conditions to reconstruct surface normals. Display photometric stereo, which employs a conventional monitor as an illumination source, has the potential to overcome limitations often encountered in bulky and difficult-to-use conventional setups. In this paper, we present differentiable display photometric stereo (DDPS), addressing an often overlooked challenge in display photometric stereo: the design of display patterns. Departing from using heuristic display patterns, DDPS learns the display patterns that yield accurate normal reconstruction for a target system in an end-to-end manner. To this end, we propose a differentiable framework that couples basis-illumination image formation with analytic photometric-stereo reconstruction. The differentiable framework facilitates the effective learning of display patterns via auto-differentiation. Also, for training supervision, we propose to use 3D printing for creating a real-world training dataset, enabling accurate reconstruction on the target real-world setup. Finally, we exploit that conventional LCD monitors emit polarized light, which allows for the optical separation of diffuse and specular reflections when combined with a polarization camera, leading to accurate normal reconstruction. Extensive evaluation of DDPS shows improved normal-reconstruction accuracy compared to heuristic patterns and demonstrates compelling properties such as robustness to pattern initialization, calibration errors, and simplifications in image formation and reconstruction.