SAFA: Structure Aware Face Animation

TL;DR

SAFA introduces a structure-aware face animation method that leverages geometric modeling with 3D morphable models and inpainting techniques to improve the realism and accuracy of generated face videos, especially with significant pose changes.

Contribution

The paper presents a novel face animation approach that incorporates geometric structures and inpainting to handle occlusions and pose variations more effectively than prior GAN-based methods.

Findings

Outperforms existing methods in realism and pose variation handling.

Effectively models facial components using 3D morphable models and affine transforms.

Demonstrates superior qualitative and quantitative results.

Abstract

Recent success of generative adversarial networks (GAN) has made great progress on the face animation task. However, the complex scene structure of a face image still makes it a challenge to generate videos with face poses significantly deviating from the source image. On one hand, without knowing the facial geometric structure, generated face images might be improperly distorted. On the other hand, some area of the generated image might be occluded in the source image, which makes it difficult for GAN to generate realistic appearance. To address these problems, we propose a structure aware face animation (SAFA) method which constructs specific geometric structures to model different components of a face image. Following the well recognized motion based face animation technique, we use a 3D morphable model (3DMM) to model the face, multiple affine transforms to model the other foreground components like hair and beard, and an identity transform to model the background. The 3DMM geometric embedding not only helps generate realistic structure for the driving scene, but also contributes to better perception of occluded area in the generated image. Besides, we further propose to exploit the widely studied inpainting technique to faithfully recover the occluded image area. Both quantitative and qualitative experiment results have shown the superiority of our method. Code is available at https://github.com/Qiulin-W/SAFA.