MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding

TL;DR

MEDTalk is a novel framework for dynamic, fine-grained emotional 3D facial animation that disentangles content and emotion, integrating multimodal inputs for realistic and controllable talking head generation.

Contribution

It introduces a disentangled embedding approach for independent control of lip movements and facial expressions, incorporating multimodal inputs for personalized, realistic emotional talking head synthesis.

Findings

Achieves synchronized lip movements with vivid emotional expressions.

Enables control over facial expressions using text and reference images.

Supports integration into industrial production pipelines.

Abstract

Audio-driven emotional 3D facial animation aims to generate synchronized lip movements and vivid facial expressions. However, most existing approaches focus on static and predefined emotion labels, limiting their diversity and naturalness. To address these challenges, we propose MEDTalk, a novel framework for fine-grained and dynamic emotional talking head generation. Our approach first disentangles content and emotion embedding spaces from motion sequences using a carefully designed cross-reconstruction process, enabling independent control over lip movements and facial expressions. Beyond conventional audio-driven lip synchronization, we integrate audio and speech text, predicting frame-wise intensity variations and dynamically adjusting static emotion features to generate realistic emotional expressions. Furthermore, to enhance control and personalization, we incorporate multimodal inputs-including text descriptions and reference expression images-to guide the generation of user-specified facial expressions. With MetaHuman as the priority, our generated results can be conveniently integrated into the industrial production pipeline. The code is available at: https://github.com/SJTU-Lucy/MEDTalk.