Learning a Delighting Prior for Facial Appearance Capture in the Wild

TL;DR

This paper introduces a novel delighting prior trained on diverse datasets to improve in-the-wild facial appearance capture, enabling high-quality reflectance estimation from casual videos.

Contribution

It proposes a dataset-conditioned delighting network with Dataset Latent Modulation, outperforming existing models and enabling automatic, high-quality facial appearance capture from casual videos.

Findings

Outperforms prior methods in reflectance estimation accuracy.

Enables automatic facial appearance capture from casual videos.

Transforms datasets into high-resolution relightable scans.

Abstract

High-quality facial appearance capture has traditionally required costly studio recording. Recent works consider an in-the-wild smartphone-based setup; however, their model-based inverse rendering paradigm struggles with the complex disentanglement of reflectance from unknown illumination. To bridge this gap, we propose to shift the paradigm into training a powerful delighting network as a prior to constrain the optimization. We leverage the OLAT dataset and the rendered Light Stage scans for training, and propose Dataset Latent Modulation (DLM) to seamlessly integrate these heterogeneous data sources. Specifically, by conditioning the core network on learnable source-aware tokens, we decouple dataset-specific styles from physical delighting principles, enabling the emergence of a delighting prior that outperforms existing proprietary models. This powerful delighting prior enables a simple and automatic appearance capture pipeline that achieves high-quality reflectance estimation from casual video inputs, outperforming prior arts by a large margin. Furthermore, we leverage our appearance capture method to transform the multi-view NeRSemble dataset into NeRSemble-Scan, a large-scale collection of 4K-resolution relightable scans. By open-sourcing our model and the NeRSemble-Scan dataset, we democratize high-end facial capture and provide a new foundation for the research community to build photorealistic digital humans.