ESGaussianFace: Emotional and Stylized Audio-Driven Facial Animation via 3D Gaussian Splatting

TL;DR

ESGaussianFace introduces a novel 3D Gaussian splatting framework for efficient, high-quality, emotionally expressive, and stylized audio-driven facial animation with 3D consistency, outperforming existing methods.

Contribution

The paper presents a new framework combining 3D Gaussian splatting, emotion-guided spatial attention, and multi-stage training for realistic emotional and stylized facial animation from audio.

Findings

Outperforms state-of-the-art in lip accuracy and expression variation

Achieves high efficiency and 3D consistency in facial animation

Effectively integrates emotion and style features for realistic results

Abstract

Most current audio-driven facial animation research primarily focuses on generating videos with neutral emotions. While some studies have addressed the generation of facial videos driven by emotional audio, efficiently generating high-quality talking head videos that integrate both emotional expressions and style features remains a significant challenge. In this paper, we propose ESGaussianFace, an innovative framework for emotional and stylized audio-driven facial animation. Our approach leverages 3D Gaussian Splatting to reconstruct 3D scenes and render videos, ensuring efficient generation of 3D consistent results. We propose an emotion-audio-guided spatial attention method that effectively integrates emotion features with audio content features. Through emotion-guided attention, the model is able to reconstruct facial details across different emotional states more accurately. To achieve emotional and stylized deformations of the 3D Gaussian points through emotion and style features, we introduce two 3D Gaussian deformation predictors. Futhermore, we propose a multi-stage training strategy, enabling the step-by-step learning of the character's lip movements, emotional variations, and style features. Our generated results exhibit high efficiency, high quality, and 3D consistency. Extensive experimental results demonstrate that our method outperforms existing state-of-the-art techniques in terms of lip movement accuracy, expression variation, and style feature expressiveness.