Learning Semantic Facial Descriptors for Accurate Face Animation

TL;DR

This paper introduces semantic facial descriptors in a learnable disentangled space to improve face animation, achieving better identity preservation and motion transfer than existing methods.

Contribution

It proposes a novel approach to decouple identity and motion in facial features using orthogonal basis vectors, enhancing face animation accuracy.

Findings

Outperforms state-of-the-art methods on VoxCeleb, HDTF, and CelebV benchmarks.

Achieves superior identity preservation in face animation.

Demonstrates improved motion transfer quality.

Abstract

Face animation is a challenging task. Existing model-based methods (utilizing 3DMMs or landmarks) often result in a model-like reconstruction effect, which doesn't effectively preserve identity. Conversely, model-free approaches face challenges in attaining a decoupled and semantically rich feature space, thereby making accurate motion transfer difficult to achieve. We introduce the semantic facial descriptors in learnable disentangled vector space to address the dilemma. The approach involves decoupling the facial space into identity and motion subspaces while endowing each of them with semantics by learning complete orthogonal basis vectors. We obtain basis vector coefficients by employing an encoder on the source and driving faces, leading to effective facial descriptors in the identity and motion subspaces. Ultimately, these descriptors can be recombined as latent codes to animate faces. Our approach successfully addresses the issue of model-based methods' limitations in high-fidelity identity and the challenges faced by model-free methods in accurate motion transfer. Extensive experiments are conducted on three challenging benchmarks (i.e. VoxCeleb, HDTF, CelebV). Comprehensive quantitative and qualitative results demonstrate that our model outperforms SOTA methods with superior identity preservation and motion transfer.