DH-FaceVid-1K: A Large-Scale High-Quality Dataset for Face Video Generation

TL;DR

This paper introduces DH-FaceVid-1K, a large-scale, high-quality face video dataset with diverse ethnic representation, supporting advanced face video generation models and benchmarks to promote fair and high-resolution face video synthesis.

Contribution

The paper presents a comprehensive, multi-ethnic face video dataset with detailed annotations, enabling improved face video generation and addressing demographic biases in existing datasets.

Findings

Established multiple face video generation models using the dataset.

Validated scaling laws for dataset size and model performance.

Enhanced diversity and quality in face video synthesis results.

Abstract

Human-centric generative models are becoming increasingly popular, giving rise to various innovative tools and applications, such as talking face videos conditioned on text or audio prompts. The core of these capabilities lies in powerful pre-trained foundation models, trained on large-scale, high-quality datasets. However, many advanced methods rely on in-house data subject to various constraints, and other current studies fail to generate high-resolution face videos, which is mainly attributed to the significant lack of large-scale, high-quality face video datasets. In this paper, we introduce a human face video dataset, \textbf{DH-FaceVid-1K}. Our collection spans 1,200 hours in total, encompassing 270,043 video clips from over 20,000 individuals. Each sample includes corresponding speech audio, facial keypoints, and text annotations. Compared to other publicly available datasets, ours distinguishes itself through its multi-ethnic coverage and high-quality, comprehensive individual attributes. We establish multiple face video generation models supporting tasks such as text-to-video and image-to-video generation. In addition, we develop comprehensive benchmarks to validate the scaling law when using different proportions of proposed dataset. Our primary aim is to contribute a face video dataset, particularly addressing the underrepresentation of Asian faces in existing curated datasets and thereby enriching the global spectrum of face-centric data and mitigating demographic biases. \textbf{Project Page:} https://luna-ai-lab.github.io/DH-FaceVid-1K/