MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation

TL;DR

MEMO introduces a memory-guided, emotion-aware diffusion approach for generating realistic, expressive talking videos with improved synchronization, identity consistency, and natural expressions, addressing key challenges in audio-driven video synthesis.

Contribution

The paper presents a novel end-to-end framework combining memory-guided temporal modeling and emotion-aware audio processing for enhanced talking video generation.

Findings

Outperforms state-of-the-art methods in quality and synchronization

Enhances long-term identity consistency and expression realism

Produces diverse, natural, and emotion-aligned talking videos

Abstract

Recent advances in video diffusion models have unlocked new potential for realistic audio-driven talking video generation. However, achieving seamless audio-lip synchronization, maintaining long-term identity consistency, and producing natural, audio-aligned expressions in generated talking videos remain significant challenges. To address these challenges, we propose Memory-guided EMOtion-aware diffusion (MEMO), an end-to-end audio-driven portrait animation approach to generate identity-consistent and expressive talking videos. Our approach is built around two key modules: (1) a memory-guided temporal module, which enhances long-term identity consistency and motion smoothness by developing memory states to store information from a longer past context to guide temporal modeling via linear attention; and (2) an emotion-aware audio module, which replaces traditional cross attention with multi-modal attention to enhance audio-video interaction, while detecting emotions from audio to refine facial expressions via emotion adaptive layer norm. Extensive quantitative and qualitative results demonstrate that MEMO generates more realistic talking videos across diverse image and audio types, outperforming state-of-the-art methods in overall quality, audio-lip synchronization, identity consistency, and expression-emotion alignment.