Lightweight Photometric Stereo for Facial Details Recovery

TL;DR

This paper introduces a lightweight neural network that uses sparse or single images under near-field lighting to accurately reconstruct 3D facial geometry, combining deep learning with photometric stereo techniques.

Contribution

It presents a novel neural network and a new dataset for photometric stereo-based 3D face reconstruction from minimal input images.

Findings

High-quality face reconstructions from one to three images.

Effective under near-field lighting conditions.

Outperforms existing methods in detail recovery.

Abstract

Recently, 3D face reconstruction from a single image has achieved great success with the help of deep learning and shape prior knowledge, but they often fail to produce accurate geometry details. On the other hand, photometric stereo methods can recover reliable geometry details, but require dense inputs and need to solve a complex optimization problem. In this paper, we present a lightweight strategy that only requires sparse inputs or even a single image to recover high-fidelity face shapes with images captured under near-field lights. To this end, we construct a dataset containing 84 different subjects with 29 expressions under 3 different lights. Data augmentation is applied to enrich the data in terms of diversity in identity, lighting, expression, etc. With this constructed dataset, we propose a novel neural network specially designed for photometric stereo based 3D face reconstruction. Extensive experiments and comparisons demonstrate that our method can generate high-quality reconstruction results with one to three facial images captured under near-field lights. Our full framework is available at https://github.com/Juyong/FacePSNet.