Facial Appearance Capture at Home with Patch-Level Reflectance Prior

TL;DR

This paper introduces a novel patch-level diffusion prior approach for improving facial reflectance capture quality from smartphone videos in dim environments, bridging the gap with studio-based methods.

Contribution

It develops a patch-level diffusion prior trained on studio scans and a posterior sampling technique to produce high-quality facial reflectance maps from casual smartphone videos.

Findings

Significantly improves reflectance quality from smartphone videos.

Closes the gap between casual and studio facial capture quality.

Enables everyday users to create detailed digital facial models.

Abstract

Existing facial appearance capture methods can reconstruct plausible facial reflectance from smartphone-recorded videos. However, the reconstruction quality is still far behind the ones based on studio recordings. This paper fills the gap by developing a novel daily-used solution with a co-located smartphone and flashlight video capture setting in a dim room. To enhance the quality, our key observation is to solve facial reflectance maps within the data distribution of studio-scanned ones. Specifically, we first learn a diffusion prior over the Light Stage scans and then steer it to produce the reflectance map that best matches the captured images. We propose to train the diffusion prior at the patch level to improve generalization ability and training stability, as current Light Stage datasets are in ultra-high resolution but limited in data size. Tailored to this prior, we propose a patch-level posterior sampling technique to sample seamless full-resolution reflectance maps from this patch-level diffusion model. Experiments demonstrate our method closes the quality gap between low-cost and studio recordings by a large margin, opening the door for everyday users to clone themselves to the digital world. Our code will be released at https://github.com/yxuhan/DoRA.