LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment

TL;DR

This paper introduces LiFT, a novel method that leverages human feedback, including rationales, to improve the alignment of text-to-video models with human preferences, resulting in better video quality and relevance.

Contribution

LiFT is the first approach to incorporate human feedback with rationales into T2V model alignment, using a new dataset and reward model to enhance video synthesis quality.

Findings

LiFT outperforms baseline models across multiple metrics.

The reward model effectively captures nuanced human preferences.

Fine-tuning with human feedback improves video relevance and quality.

Abstract

Recent advances in text-to-video (T2V) generative models have shown impressive capabilities. However, these models are still inadequate in aligning synthesized videos with human preferences (e.g., accurately reflecting text descriptions), which is particularly difficult to address, as human preferences are subjective and challenging to formalize as objective functions. Existing studies train video quality assessment models that rely on human-annotated ratings for video evaluation but overlook the reasoning behind evaluations, limiting their ability to capture nuanced human criteria. Moreover, aligning T2V model using video-based human feedback remains unexplored. Therefore, this paper proposes LiFT, the first method designed to leverage human feedback for T2V model alignment. Specifically, we first construct a Human Rating Annotation dataset, LiFT-HRA, consisting of approximately 10k human annotations, each including a score and its corresponding rationale. Based on this, we train a reward model LiFT-Critic to learn reward function effectively, which serves as a proxy for human judgment, measuring the alignment between given videos and human expectations. Lastly, we leverage the learned reward function to align the T2V model by maximizing the reward-weighted likelihood. As a case study, we apply our pipeline to CogVideoX-2B, showing that the fine-tuned model outperforms the CogVideoX-5B across all 16 metrics, highlighting the potential of human feedback in improving the alignment and quality of synthesized videos.