Multimodal Diffusion Transformer with Memory Bank for Scalable Long-Duration Talking Video Generation

TL;DR

LetsTalk is a diffusion transformer framework with a memory bank for scalable, high-quality, long-duration talking video generation, addressing issues like visual degradation and temporal artifacts.

Contribution

It introduces a noise-regularized memory bank and a spatiotemporal-aware transformer, improving long-video synthesis quality and efficiency with novel fusion schemes.

Findings

Achieves state-of-the-art quality in long talking video synthesis.

Produces temporally coherent and diverse videos with fewer parameters.

Outperforms previous methods in realism and efficiency.

Abstract

Long-duration talking video synthesis faces enduring challenges in achieving high video quality, portrait consistency, temporal coherence, and computational efficiency. As video length increases, issues such as visual degradation, portrait drift, temporal artifacts, and error accumulation become increasingly problematic, severely affecting the realism and reliability of the results. To address these challenges, we present LetsTalk, a diffusion transformer framework equipped with multimodal guidance and a novel memory bank mechanism, explicitly maintaining contextual continuity and enabling robust, high-quality, and efficient generation of long-duration talking videos. In particular, LetsTalk introduces a noise-regularized memory bank to alleviate error accumulation and sampling artifacts during extended video generation. To further improve efficiency and spatiotemporal modeling, LetsTalk employs a deep compression autoencoder and a spatiotemporal-aware transformer with linear attention for effective multimodal fusion. We systematically analyze three fusion schemes and show that combining deep (Symbiotic Fusion) for portrait features and shallow (Direct Fusion) for audio achieves superior visual realism and precise speech-driven motion, while preserving diversity of movements. Extensive experiments demonstrate that LetsTalk establishes new state-of-the-art in generation quality, producing temporally coherent and realistic talking videos with enhanced diversity and liveliness, and maintains remarkable efficiency with 8x fewer parameters than previous approaches.