TalkVid: A Large-Scale Diversified Dataset for Audio-Driven Talking Head Synthesis

TL;DR

TalkVid introduces a large, diverse, high-quality dataset for audio-driven talking head synthesis, significantly improving model generalization across ethnicity, language, and age groups, and highlighting subgroup performance disparities.

Contribution

The paper presents TalkVid, a large-scale, meticulously curated dataset that enhances diversity and quality for training talking head models, and introduces TalkVid-Bench for comprehensive evaluation.

Findings

Models trained on TalkVid outperform previous datasets.

TalkVid improves cross-dataset generalization.

Performance disparities across demographic subgroups are revealed.

Abstract

Audio-driven talking head synthesis has achieved remarkable photorealism, yet state-of-the-art (SOTA) models exhibit a critical failure: they lack generalization to the full spectrum of human diversity in ethnicity, language, and age groups. We argue that this generalization gap is a direct symptom of limitations in existing training data, which lack the necessary scale, quality, and diversity. To address this challenge, we introduce TalkVid, a new large-scale, high-quality, and diverse dataset containing 1244 hours of video from 7729 unique speakers. TalkVid is curated through a principled, multi-stage automated pipeline that rigorously filters for motion stability, aesthetic quality, and facial detail, and is validated against human judgments to ensure its reliability. Furthermore, we construct and release TalkVid-Bench, a stratified evaluation set of 500 clips meticulously balanced across key demographic and linguistic axes. Our experiments demonstrate that a model trained on TalkVid outperforms counterparts trained on previous datasets, exhibiting superior cross-dataset generalization. Crucially, our analysis on TalkVid-Bench reveals performance disparities across subgroups that are obscured by traditional aggregate metrics, underscoring its necessity for future research. Code and data can be found in https://github.com/FreedomIntelligence/TalkVid