G3FA: Geometry-guided GAN for Face Animation

TL;DR

G3FA introduces a geometry-guided GAN that incorporates 3D facial information from 2D images to enhance face animation quality and consistency in real-time reenactment tasks.

Contribution

The paper presents a novel approach integrating 3D geometry extraction and inverse rendering into GAN-based face animation, improving realism and geometry consistency.

Findings

Outperforms state-of-the-art methods on VoxCeleb2 and TalkingHead benchmarks.

Enhances geometry consistency and visual quality in face reenactment.

Effective use of 3D information from 2D images improves animation realism.

Abstract

Animating human face images aims to synthesize a desired source identity in a natural-looking way mimicking a driving video's facial movements. In this context, Generative Adversarial Networks have demonstrated remarkable potential in real-time face reenactment using a single source image, yet are constrained by limited geometry consistency compared to graphic-based approaches. In this paper, we introduce Geometry-guided GAN for Face Animation (G3FA) to tackle this limitation. Our novel approach empowers the face animation model to incorporate 3D information using only 2D images, improving the image generation capabilities of the talking head synthesis model. We integrate inverse rendering techniques to extract 3D facial geometry properties, improving the feedback loop to the generator through a weighted average ensemble of discriminators. In our face reenactment model, we leverage 2D motion warping to capture motion dynamics along with orthogonal ray sampling and volume rendering techniques to produce the ultimate visual output. To evaluate the performance of our G3FA, we conducted comprehensive experiments using various evaluation protocols on VoxCeleb2 and TalkingHead benchmarks to demonstrate the effectiveness of our proposed framework compared to the state-of-the-art real-time face animation methods.