EmoDiffTalk:Emotion-aware Diffusion for Editable 3D Gaussian Talking Head

TL;DR

EmoDiffTalk introduces an emotion-aware diffusion framework for editable 3D Gaussian talking heads, enabling fine-grained, multimodal emotional editing with high fidelity and controllability, advancing the realism and expressiveness of 3D talking head synthesis.

Contribution

It presents a novel emotion-aware Gaussian diffusion method with an AU-based emotion controller for high-quality, editable 3D talking heads with continuous emotional expression editing.

Findings

Superior emotional subtlety demonstrated on datasets

High lip-sync fidelity achieved

Enhanced controllability over emotional expressions

Abstract

Recent photo-realistic 3D talking head via 3D Gaussian Splatting still has significant shortcoming in emotional expression manipulation, especially for fine-grained and expansive dynamics emotional editing using multi-modal control. This paper introduces a new editable 3D Gaussian talking head, i.e. EmoDiffTalk. Our key idea is a novel Emotion-aware Gaussian Diffusion, which includes an action unit (AU) prompt Gaussian diffusion process for fine-grained facial animator, and moreover an accurate text-to-AU emotion controller to provide accurate and expansive dynamic emotional editing using text input. Experiments on public EmoTalk3D and RenderMe-360 datasets demonstrate superior emotional subtlety, lip-sync fidelity, and controllability of our EmoDiffTalk over previous works, establishing a principled pathway toward high-quality, diffusion-driven, multimodal editable 3D talking-head synthesis. To our best knowledge, our EmoDiffTalk is one of the first few 3D Gaussian Splatting talking-head generation framework, especially supporting continuous, multimodal emotional editing within the AU-based expression space.