VDOT: Efficient Unified Video Creation via Optimal Transport Distillation

TL;DR

VDOT introduces an efficient, unified video creation model using optimal transport distillation, significantly reducing generation time while maintaining high quality, and providing a standardized benchmark for evaluation.

Contribution

The paper proposes a novel optimal transport-based distillation method for unified video creation, improving efficiency and stability over traditional KL-based approaches.

Findings

VDOT outperforms baselines with fewer steps

Achieves comparable quality with 100 steps using only 4 steps

Provides a new benchmark for unified video creation evaluation

Abstract

The rapid development of generative models has significantly advanced image and video applications. Among these, video creation, aimed at generating videos under various conditions, has gained substantial attention. However, existing video creation models either focus solely on a few specific conditions or suffer from excessively long generation times due to complex model inference, making them impractical for real-world applications. To mitigate these issues, we propose an efficient unified video creation model, named VDOT. Concretely, we model the training process with the distribution matching distillation (DMD) paradigm. Instead of using the Kullback-Leibler (KL) minimization, we additionally employ a novel computational optimal transport (OT) technique to optimize the discrepancy between the real and fake score distributions. The OT distance inherently imposes geometric constraints, mitigating potential zero-forcing or gradient collapse issues that may arise during KL-based distillation within the few-step generation scenario, and thus, enhances the efficiency and stability of the distillation process. Further, we integrate a discriminator to enable the model to perceive real video data, thereby enhancing the quality of generated videos. To support training unified video creation models, we propose a fully automated pipeline for video data annotation and filtering that accommodates multiple video creation tasks. Meanwhile, we curate a unified testing benchmark, UVCBench, to standardize evaluation. Experiments demonstrate that our 4-step VDOT outperforms or matches other baselines with 100 denoising steps.