GaFET: Learning Geometry-aware Facial Expression Translation from In-The-Wild Images

TL;DR

GaFET introduces a geometry-aware framework for facial expression translation from in-the-wild images, utilizing 3D representations and a novel transformer to improve realism and generalization without requiring video data or annotations.

Contribution

The paper proposes a novel GaFET framework combining 3D facial modeling, a multi-level feature aligned transformer, and a StyleGAN-based de-expression model for improved expression transfer.

Findings

Achieves higher-quality expression transfer than state-of-the-art methods.

Handles complex poses and textures effectively.

Operates without video data or annotated training sets.

Abstract

While current face animation methods can manipulate expressions individually, they suffer from several limitations. The expressions manipulated by some motion-based facial reenactment models are crude. Other ideas modeled with facial action units cannot generalize to arbitrary expressions not covered by annotations. In this paper, we introduce a novel Geometry-aware Facial Expression Translation (GaFET) framework, which is based on parametric 3D facial representations and can stably decoupled expression. Among them, a Multi-level Feature Aligned Transformer is proposed to complement non-geometric facial detail features while addressing the alignment challenge of spatial features. Further, we design a De-expression model based on StyleGAN, in order to reduce the learning difficulty of GaFET in unpaired "in-the-wild" images. Extensive qualitative and quantitative experiments demonstrate that we achieve higher-quality and more accurate facial expression transfer results compared to state-of-the-art methods, and demonstrate applicability of various poses and complex textures. Besides, videos or annotated training data are omitted, making our method easier to use and generalize.