3DFaceFill: An Analysis-By-Synthesis Approach to Face Completion

TL;DR

3DFaceFill introduces an analysis-by-synthesis method for face completion that explicitly models 3D face geometry and appearance, resulting in more realistic and consistent completions under large pose, shape, and illumination variations.

Contribution

It proposes a novel 3D-aware face completion framework that disentangles facial factors and operates on UV representations for improved realism and geometric consistency.

Findings

Up to 4dB higher PSNR compared to state-of-the-art.

25% better LPIPS for large masks.

More photorealistic and geometrically consistent face completions.

Abstract

Existing face completion solutions are primarily driven by end-to-end models that directly generate 2D completions of 2D masked faces. By having to implicitly account for geometric and photometric variations in facial shape and appearance, such approaches result in unrealistic completions, especially under large variations in pose, shape, illumination and mask sizes. To alleviate these limitations, we introduce 3DFaceFill, an analysis-by-synthesis approach for face completion that explicitly considers the image formation process. It comprises three components, (1) an encoder that disentangles the face into its constituent 3D mesh, 3D pose, illumination and albedo factors, (2) an autoencoder that inpaints the UV representation of facial albedo, and (3) a renderer that resynthesizes the completed face. By operating on the UV representation, 3DFaceFill affords the power of correspondence and allows us to naturally enforce geometrical priors (e.g. facial symmetry) more effectively. Quantitatively, 3DFaceFill improves the state-of-the-art by up to 4dB higher PSNR and 25% better LPIPS for large masks. And, qualitatively, it leads to demonstrably more photorealistic face completions over a range of masks and occlusions while preserving consistency in global and component-wise shape, pose, illumination and eye-gaze.