DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation

TL;DR

DEEPTalk is a novel speech-driven 3D facial animation method that generates emotionally rich, diverse, and realistic facial expressions by leveraging probabilistic emotion embeddings and hierarchical motion priors, improving over monotonous previous approaches.

Contribution

The paper introduces DEEPTalk, combining probabilistic contrastive learning for emotion embedding with hierarchical VQ-VAE for dynamic facial motion, enabling emotionally expressive and diverse 3D face animations from speech.

Findings

Produces diverse, emotionally rich facial animations

Maintains accurate lip-sync in generated faces

Outperforms existing methods in realism and expressiveness

Abstract

Speech-driven 3D facial animation has garnered lots of attention thanks to its broad range of applications. Despite recent advancements in achieving realistic lip motion, current methods fail to capture the nuanced emotional undertones conveyed through speech and produce monotonous facial motion. These limitations result in blunt and repetitive facial animations, reducing user engagement and hindering their applicability. To address these challenges, we introduce DEEPTalk, a novel approach that generates diverse and emotionally rich 3D facial expressions directly from speech inputs. To achieve this, we first train DEE (Dynamic Emotion Embedding), which employs probabilistic contrastive learning to forge a joint emotion embedding space for both speech and facial motion. This probabilistic framework captures the uncertainty in interpreting emotions from speech and facial motion, enabling the derivation of emotion vectors from its multifaceted space. Moreover, to generate dynamic facial motion, we design TH-VQVAE (Temporally Hierarchical VQ-VAE) as an expressive and robust motion prior overcoming limitations of VAEs and VQ-VAEs. Utilizing these strong priors, we develop DEEPTalk, a talking head generator that non-autoregressively predicts codebook indices to create dynamic facial motion, incorporating a novel emotion consistency loss. Extensive experiments on various datasets demonstrate the effectiveness of our approach in creating diverse, emotionally expressive talking faces that maintain accurate lip-sync. Our project page is available at https://whwjdqls.github.io/deeptalk\_website/