WildCap: Facial Albedo Capture in the Wild via Hybrid Inverse Rendering

TL;DR

WildCap introduces a hybrid inverse rendering approach that enables high-quality facial albedo capture from smartphone videos in natural settings, overcoming lighting complexities and local artifacts to match controlled environment quality.

Contribution

The paper presents a novel hybrid inverse rendering framework with a texel grid lighting model and diffusion prior optimization, advancing in-the-wild facial albedo capture technology.

Findings

Significantly improves in-the-wild facial albedo quality.

Reduces artifacts like shadow-baking in albedo predictions.

Closes the quality gap between in-the-wild and controlled captures.

Abstract

Existing methods achieve high-quality facial albedo capture under controllable lighting, which increases capture cost and limits usability. We propose WildCap, a novel method for high-quality facial albedo capture from a smartphone video recorded in the wild. To disentangle high-quality albedo from complex lighting effects in in-the-wild captures, we propose a novel hybrid inverse rendering framework. We first apply a data-driven method, i.e., SwitchLight, to convert the captured images into more constrained conditions and then adopt model-based inverse rendering. However, unavoidable local artifacts in network predictions, such as shadow-baking, are non-physical and thus hinder accurate inverse rendering of lighting and material. To address this, we propose a novel texel grid lighting model to explain non-physical effects as clean albedo illuminated by local physical lighting. During optimization, we jointly sample a diffusion prior for the albedo map and optimize the lighting, effectively resolving scale ambiguity between local lights and albedo. Other reflectance maps are then predicted from the albedo. Our method achieves significantly better results than prior arts in the same capture setup, closing the quality gap between in-the-wild and controllable recordings by a large margin.