TalkVerse: Democratizing Minute-Long Audio-Driven Video Generation

TL;DR

TalkVerse introduces a large, open dataset and a scalable model for minute-long, audio-driven talking video generation, enabling fair comparison and advancing research with lower inference costs.

Contribution

It provides a comprehensive, open dataset and a reproducible baseline model for audio-driven talking video generation, facilitating fair benchmarking and reducing computational barriers.

Findings

Achieves high-quality, minute-long video generation with low drift.

Delivers lip-sync and visual quality comparable to larger models.

Supports zero-shot video dubbing with controlled latent noise.

Abstract

We introduce TalkVerse, a large-scale, open corpus for single-person, audio-driven talking video generation designed to enable fair, reproducible comparison across methods. While current state-of-the-art systems rely on closed data or compute-heavy models, TalkVerse offers 2.3 million high-resolution (720p/1080p) audio-video synchronized clips totaling 6.3k hours. These are curated from over 60k hours of video via a transparent pipeline that includes scene-cut detection, aesthetic assessment, strict audio-visual synchronization checks, and comprehensive annotations including 2D skeletons and structured visual/audio-style captions. Leveraging TalkVerse, we present a reproducible 5B DiT baseline built on Wan2.2-5B. By utilizing a video VAE with a high downsampling ratio and a sliding window mechanism with motion-frame context, our model achieves minute-long generation with low drift. It delivers comparable lip-sync and visual quality to the 14B Wan-S2V model but with 10$\times$ lower inference cost. To enhance storytelling in long videos, we integrate an MLLM director to rewrite prompts based on audio and visual cues. Furthermore, our model supports zero-shot video dubbing via controlled latent noise injection. We open-source the dataset, training recipes, and 5B checkpoints to lower barriers for research in audio-driven human video generation. Project Page: https://zhenzhiwang.github.io/talkverse/