3D GAN Inversion with Facial Symmetry Prior

TL;DR

This paper introduces a 3D GAN inversion method that incorporates facial symmetry prior and depth-guided supervision to improve 3D reconstruction accuracy and texture quality from monocular images.

Contribution

It proposes a novel facial symmetry prior and constraints for 3D GAN inversion, enhancing geometry robustness and texture fidelity in monocular 3D portrait reconstruction.

Findings

Improved 3D shape accuracy in reconstructions.

Enhanced texture quality in novel viewpoints.

Superior performance in image editing tasks.

Abstract

Recently, a surge of high-quality 3D-aware GANs have been proposed, which leverage the generative power of neural rendering. It is natural to associate 3D GANs with GAN inversion methods to project a real image into the generator's latent space, allowing free-view consistent synthesis and editing, referred as 3D GAN inversion. Although with the facial prior preserved in pre-trained 3D GANs, reconstructing a 3D portrait with only one monocular image is still an ill-pose problem. The straightforward application of 2D GAN inversion methods focuses on texture similarity only while ignoring the correctness of 3D geometry shapes. It may raise geometry collapse effects, especially when reconstructing a side face under an extreme pose. Besides, the synthetic results in novel views are prone to be blurry. In this work, we propose a novel method to promote 3D GAN inversion by introducing facial symmetry prior. We design a pipeline and constraints to make full use of the pseudo auxiliary view obtained via image flipping, which helps obtain a robust and reasonable geometry shape during the inversion process. To enhance texture fidelity in unobserved viewpoints, pseudo labels from depth-guided 3D warping can provide extra supervision. We design constraints aimed at filtering out conflict areas for optimization in asymmetric situations. Comprehensive quantitative and qualitative evaluations on image reconstruction and editing demonstrate the superiority of our method.