EDTalk: Efficient Disentanglement for Emotional Talking Head Synthesis

TL;DR

EDTalk introduces a novel framework for disentangled control of facial motions in talking head synthesis, enabling independent manipulation of mouth, pose, and expression conditioned on audio or video inputs, with improved efficiency and flexibility.

Contribution

The paper proposes a lightweight, orthogonality-enforced disentanglement framework that allows independent control of facial features and shared priors for audio-driven synthesis.

Findings

Effective disentanglement of facial motions demonstrated

Independent control of mouth, pose, and expression achieved

Enhanced synthesis quality and flexibility

Abstract

Achieving disentangled control over multiple facial motions and accommodating diverse input modalities greatly enhances the application and entertainment of the talking head generation. This necessitates a deep exploration of the decoupling space for facial features, ensuring that they a) operate independently without mutual interference and b) can be preserved to share with different modal input, both aspects often neglected in existing methods. To address this gap, this paper proposes a novel Efficient Disentanglement framework for Talking head generation (EDTalk). Our framework enables individual manipulation of mouth shape, head pose, and emotional expression, conditioned on video or audio inputs. Specifically, we employ three lightweight modules to decompose the facial dynamics into three distinct latent spaces representing mouth, pose, and expression, respectively. Each space is characterized by a set of learnable bases whose linear combinations define specific motions. To ensure independence and accelerate training, we enforce orthogonality among bases and devise an efficient training strategy to allocate motion responsibilities to each space without relying on external knowledge. The learned bases are then stored in corresponding banks, enabling shared visual priors with audio input. Furthermore, considering the properties of each space, we propose an Audio-to-Motion module for audio-driven talking head synthesis. Experiments are conducted to demonstrate the effectiveness of EDTalk. We recommend watching the project website: https://tanshuai0219.github.io/EDTalk/