Beyond the Golden Data: Resolving the Motion-Vision Quality Dilemma via Timestep Selective Training

TL;DR

This paper introduces Timestep-aware Quality Decoupling (TQD), a novel training method that addresses the Motion-Vision Quality Dilemma in video data by selectively sampling data at different training timesteps, improving video generation models.

Contribution

It proposes a new data sampling strategy that decouples quality and motion aspects during training, enabling models to learn effectively from imbalanced data.

Findings

TQD outperforms traditional training on high-quality data.

Training on separated imbalanced data can surpass conventional methods.

TQD improves model performance across various data scenarios.

Abstract

Recent advances in video generation models have achieved impressive results. However, these models heavily rely on the use of high-quality data that combines both high visual quality and high motion quality. In this paper, we identify a key challenge in video data curation: the Motion-Vision Quality Dilemma. We discovered that visual quality and motion intensity inherently exhibit a negative correlation, making it hard to obtain golden data that excels in both aspects. To address this challenge, we first examine the hierarchical learning dynamics of video diffusion models and conduct gradient-based analysis on quality-degraded samples. We discover that quality-imbalanced data can produce gradients similar to golden data at appropriate timesteps. Based on this, we introduce the novel concept of Timestep selection in Training Process. We propose Timestep-aware Quality Decoupling (TQD), which modifies the data sampling distribution to better match the model's learning process. For certain types of data, the sampling distribution is skewed toward higher timesteps for motion-rich data, while high visual quality data is more likely to be sampled during lower timesteps. Through extensive experiments, we demonstrate that TQD enables training exclusively on separated imbalanced data to achieve performance surpassing conventional training with better data, challenging the necessity of perfect data in video generation. Moreover, our method also boosts model performance when trained on high-quality data, showcasing its effectiveness across different data scenarios.