EAD-Net: Emotion-Aware Talking Head Generation with Spatial Refinement and Temporal Coherence

TL;DR

EAD-Net is a novel emotion-aware talking head generation framework that improves lip-sync, emotional expression, and temporal coherence in long videos using advanced spatio-temporal modeling and semantic guidance.

Contribution

The paper introduces a diffusion model-based network with SyncNet supervision, TREPA, STDA, TFRM, and semantic guidance to enhance expressive, coherent, and accurate talking head videos.

Findings

Outperforms existing methods in lip-sync accuracy.

Achieves better temporal consistency in long videos.

Enhances emotional expression accuracy.

Abstract

Emotionally talking head video generation aims to generate expressive portrait videos with accurate lip synchronization and emotional facial expressions. Current methods rely on simple emotional labels, leading to insufficient semantic information. While introducing high-level semantics enhances expressiveness, it easily causes lip-sync degradation. Furthermore, mainstream generation methods struggle to balance computational efficiency and global motion awareness in long videos and suffer from poor temporal coherence. Therefore, we propose an \textbf{E}motion-\textbf{A}ware \textbf{D}iffusion model-based \textbf{Net}work, called \textbf{EAD-Net}. We introduce SyncNet supervision and Temporal Representation Alignment (TREPA) to mitigate lip-sync degradation caused by multi-modal fusion. To model complex spatio-temporal dependencies in long video sequences, we propose a Spatio-Temporal Directional Attention (STDA) mechanism that captures global motion patterns through strip attention. Additionally, we design a Temporal Frame graph Reasoning Module (TFRM) to explicitly model temporal coherence between video frames through graph structure learning. To enhance emotional semantic control, a large language model is employed to extract textual descriptions from real videos, serving as high-level semantic guidance. Experiments on the HDTF and MEAD datasets demonstrate that our method outperforms existing methods in terms of lip-sync accuracy, temporal consistency, and emotional accuracy.