Efficient Training for Human Video Generation with Entropy-Guided Prioritized Progressive Learning

TL;DR

This paper presents Ent-Prog, an efficient training framework for human video diffusion models that reduces training time and memory usage by prioritizing critical components and adaptively increasing complexity, without sacrificing quality.

Contribution

It introduces Conditional Entropy Inflation and an adaptive progressive schedule to optimize training efficiency for human video generation models.

Findings

Achieves up to 2.2× training speedup

Reduces GPU memory by 2.4×

Maintains high-quality video generation

Abstract

Human video generation has advanced rapidly with the development of diffusion models, but the high computational cost and substantial memory consumption associated with training these models on high-resolution, multi-frame data pose significant challenges. In this paper, we propose Entropy-Guided Prioritized Progressive Learning (Ent-Prog), an efficient training framework tailored for diffusion models on human video generation. First, we introduce Conditional Entropy Inflation (CEI) to assess the importance of different model components on the target conditional generation task, enabling prioritized training of the most critical components. Second, we introduce an adaptive progressive schedule that adaptively increases computational complexity during training by measuring the convergence efficiency. Ent-Prog reduces both training time and GPU memory consumption while maintaining model performance. Extensive experiments across three datasets, demonstrate the effectiveness of Ent-Prog, achieving up to 2.2$\times$ training speedup and 2.4$\times$ GPU memory reduction without compromising generative performance.