EmoFace: Emotion-Content Disentangled Speech-Driven 3D Talking Face Animation

TL;DR

EmoFace is a novel two-stream network that disentangles emotion and content in speech-driven 3D facial animation, using Mesh Attention and a self-growing training scheme to improve realism and emotional expression.

Contribution

The paper introduces EmoFace, the first to incorporate emotion-content disentanglement with Mesh Attention and a self-growing training scheme for enhanced 3D facial animation.

Findings

Achieves state-of-the-art performance on 3D-RAVDESS and VOCASET datasets.

Effectively captures emotional expressions and lip synchronization.

Outperforms existing methods in quantitative and qualitative evaluations.

Abstract

The creation of increasingly vivid 3D talking face has become a hot topic in recent years. Currently, most speech-driven works focus on lip synchronisation but neglect to effectively capture the correlations between emotions and facial motions. To address this problem, we propose a two-stream network called EmoFace, which consists of an emotion branch and a content branch. EmoFace employs a novel Mesh Attention mechanism to analyse and fuse the emotion features and content features. Particularly, a newly designed spatio-temporal graph-based convolution, SpiralConv3D, is used in Mesh Attention to learn potential temporal and spatial feature dependencies between mesh vertices. In addition, to the best of our knowledge, it is the first time to introduce a new self-growing training scheme with intermediate supervision to dynamically adjust the ratio of groundtruth adopted in the 3D face animation task. Comprehensive quantitative and qualitative evaluations on our high-quality 3D emotional facial animation dataset, 3D-RAVDESS ($4.8863\times 10^{-5}$mm for LVE and $0.9509\times 10^{-5}$mm for EVE), together with the public dataset VOCASET ($2.8669\times 10^{-5}$mm for LVE and $0.4664\times 10^{-5}$mm for EVE), demonstrate that our approach achieves state-of-the-art performance.