HuViDPO:Enhancing Video Generation through Direct Preference Optimization for Human-Centric Alignment

TL;DR

HuViDPO introduces a novel approach integrating Direct Preference Optimization into text-to-video generation, improving human alignment and video quality through a structured loss function, preference datasets, and a first-frame-conditioned strategy.

Contribution

This work is the first to apply DPO to T2V tasks, developing a new loss function, constructing preference datasets, and employing a first-frame-conditioned strategy for better video generation.

Findings

Enhanced alignment of generated videos with human preferences.

Improved video quality with reduced training costs.

Flexible video generation guided by initial frame information.

Abstract

With the rapid development of AIGC technology, significant progress has been made in diffusion model-based technologies for text-to-image (T2I) and text-to-video (T2V). In recent years, a few studies have introduced the strategy of Direct Preference Optimization (DPO) into T2I tasks, significantly enhancing human preferences in generated images. However, existing T2V generation methods lack a well-formed pipeline with exact loss function to guide the alignment of generated videos with human preferences using DPO strategies. Additionally, challenges such as the scarcity of paired video preference data hinder effective model training. At the same time, the lack of training datasets poses a risk of insufficient flexibility and poor video generation quality in the generated videos. Based on those problems, our work proposes three targeted solutions in sequence. 1) Our work is the first to introduce the DPO strategy into the T2V tasks. By deriving a carefully structured loss function, we utilize human feedback to align video generation with human preferences. We refer to this new method as HuViDPO. 2) Our work constructs small-scale human preference datasets for each action category and fine-tune this model, improving the aesthetic quality of the generated videos while reducing training costs. 3) We adopt a First-Frame-Conditioned strategy, leveraging the rich in formation from the first frame to guide the generation of subsequent frames, enhancing flexibility in video generation. At the same time, we employ a SparseCausal Attention mechanism to enhance the quality of the generated videos.More details and examples can be accessed on our website: https://tankowa.github.io/HuViDPO. github.io/.