T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design

TL;DR

This paper introduces T2V-Turbo-v2, a post-training enhancement for text-to-video diffusion models that leverages data, reward feedback, and conditional guidance to improve video quality and motion accuracy, achieving state-of-the-art results.

Contribution

The paper presents a novel post-training distillation method integrating supervision signals and conditional guidance to significantly improve T2V models.

Findings

Achieves new state-of-the-art on VBench with a score of 85.13.

Effectively improves motion quality in generated videos.

Demonstrates the importance of dataset tailoring and reward models.

Abstract

In this paper, we focus on enhancing a diffusion-based text-to-video (T2V) model during the post-training phase by distilling a highly capable consistency model from a pretrained T2V model. Our proposed method, T2V-Turbo-v2, introduces a significant advancement by integrating various supervision signals, including high-quality training data, reward model feedback, and conditional guidance, into the consistency distillation process. Through comprehensive ablation studies, we highlight the crucial importance of tailoring datasets to specific learning objectives and the effectiveness of learning from diverse reward models for enhancing both the visual quality and text-video alignment. Additionally, we highlight the vast design space of conditional guidance strategies, which centers on designing an effective energy function to augment the teacher ODE solver. We demonstrate the potential of this approach by extracting motion guidance from the training datasets and incorporating it into the ODE solver, showcasing its effectiveness in improving the motion quality of the generated videos with the improved motion-related metrics from VBench and T2V-CompBench. Empirically, our T2V-Turbo-v2 establishes a new state-of-the-art result on VBench, with a Total score of 85.13, surpassing proprietary systems such as Gen-3 and Kling.