EmoTalkingGaussian: Continuous Emotion-conditioned Talking Head Synthesis

TL;DR

EmoTalkingGaussian is a novel 3D Gaussian splatting-based talking head synthesis model that can generate emotionally expressive, lip-synchronized talking heads conditioned on continuous emotion values, trained with self-supervised methods for in-the-wild audio.

Contribution

It introduces a lip-aligned emotional face generator and a self-supervised training approach for emotion-conditioned talking head synthesis with improved realism and synchronization.

Findings

Outperforms state-of-the-art in image quality metrics.

Achieves better emotion expression accuracy.

Enhances lip synchronization in wild audio scenarios.

Abstract

3D Gaussian splatting-based talking head synthesis has recently gained attention for its ability to render high-fidelity images with real-time inference speed. However, since it is typically trained on only a short video that lacks the diversity in facial emotions, the resultant talking heads struggle to represent a wide range of emotions. To address this issue, we propose a lip-aligned emotional face generator and leverage it to train our EmoTalkingGaussian model. It is able to manipulate facial emotions conditioned on continuous emotion values (i.e., valence and arousal); while retaining synchronization of lip movements with input audio. Additionally, to achieve the accurate lip synchronization for in-the-wild audio, we introduce a self-supervised learning method that leverages a text-to-speech network and a visual-audio synchronization network. We experiment our EmoTalkingGaussian on publicly available videos and have obtained better results than state-of-the-arts in terms of image quality (measured in PSNR, SSIM, LPIPS), emotion expression (measured in V-RMSE, A-RMSE, V-SA, A-SA, Emotion Accuracy), and lip synchronization (measured in LMD, Sync-E, Sync-C), respectively.