3D-FM GAN: Towards 3D-Controllable Face Manipulation

TL;DR

The paper introduces 3D-FM GAN, a new framework for high-quality, identity-preserving, 3D-controllable face manipulation that outperforms prior methods in editability, realism, and generalizability without post-training tuning.

Contribution

It proposes a novel conditional GAN architecture with specialized training strategies and a multiplicative co-modulation design for efficient 3D face editing.

Findings

Outperforms prior methods in editability and realism

Maintains high identity preservation during manipulation

Demonstrates strong generalization to large pose and out-of-domain images

Abstract

3D-controllable portrait synthesis has significantly advanced, thanks to breakthroughs in generative adversarial networks (GANs). However, it is still challenging to manipulate existing face images with precise 3D control. While concatenating GAN inversion and a 3D-aware, noise-to-image GAN is a straight-forward solution, it is inefficient and may lead to noticeable drop in editing quality. To fill this gap, we propose 3D-FM GAN, a novel conditional GAN framework designed specifically for 3D-controllable face manipulation, and does not require any tuning after the end-to-end learning phase. By carefully encoding both the input face image and a physically-based rendering of 3D edits into a StyleGAN's latent spaces, our image generator provides high-quality, identity-preserved, 3D-controllable face manipulation. To effectively learn such novel framework, we develop two essential training strategies and a novel multiplicative co-modulation architecture that improves significantly upon naive schemes. With extensive evaluations, we show that our method outperforms the prior arts on various tasks, with better editability, stronger identity preservation, and higher photo-realism. In addition, we demonstrate a better generalizability of our design on large pose editing and out-of-domain images.