OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation

TL;DR

OpenHumanVid introduces a large, high-quality human-centric video dataset with detailed annotations, significantly improving the training and quality of human video generation models through enhanced data and alignment strategies.

Contribution

The paper presents a new large-scale dataset with detailed human-centric annotations and extends diffusion transformer models to improve human video generation.

Findings

Large-scale dataset improves evaluation metrics for human video generation.

Proper alignment of text with human appearance and motion is crucial for quality.

Extended models trained on the dataset outperform previous methods.

Abstract

Recent advancements in visual generation technologies have markedly increased the scale and availability of video datasets, which are crucial for training effective video generation models. However, a significant lack of high-quality, human-centric video datasets presents a challenge to progress in this field. To bridge this gap, we introduce OpenHumanVid, a large-scale and high-quality human-centric video dataset characterized by precise and detailed captions that encompass both human appearance and motion states, along with supplementary human motion conditions, including skeleton sequences and speech audio. To validate the efficacy of this dataset and the associated training strategies, we propose an extension of existing classical diffusion transformer architectures and conduct further pretraining of our models on the proposed dataset. Our findings yield two critical insights: First, the incorporation of a large-scale, high-quality dataset substantially enhances evaluation metrics for generated human videos while preserving performance in general video generation tasks. Second, the effective alignment of text with human appearance, human motion, and facial motion is essential for producing high-quality video outputs. Based on these insights and corresponding methodologies, the straightforward extended network trained on the proposed dataset demonstrates an obvious improvement in the generation of human-centric videos. Project page https://fudan-generative-vision.github.io/OpenHumanVid