InstructVideo: Instructing Video Diffusion Models with Human Feedback

TL;DR

InstructVideo introduces a human feedback-based fine-tuning approach for text-to-video diffusion models, improving video quality by efficient reward-based editing and repurposing image reward models for better alignment with human preferences.

Contribution

The paper presents a novel reward fine-tuning method that reduces computational costs and leverages image reward models for improved video generation quality.

Findings

Enhanced video quality with human feedback fine-tuning

Reduced fine-tuning computational cost through partial inference

Effective use of image reward models for video preference alignment

Abstract

Diffusion models have emerged as the de facto paradigm for video generation. However, their reliance on web-scale data of varied quality often yields results that are visually unappealing and misaligned with the textual prompts. To tackle this problem, we propose InstructVideo to instruct text-to-video diffusion models with human feedback by reward fine-tuning. InstructVideo has two key ingredients: 1) To ameliorate the cost of reward fine-tuning induced by generating through the full DDIM sampling chain, we recast reward fine-tuning as editing. By leveraging the diffusion process to corrupt a sampled video, InstructVideo requires only partial inference of the DDIM sampling chain, reducing fine-tuning cost while improving fine-tuning efficiency. 2) To mitigate the absence of a dedicated video reward model for human preferences, we repurpose established image reward models, e.g., HPSv2. To this end, we propose Segmental Video Reward, a mechanism to provide reward signals based on segmental sparse sampling, and Temporally Attenuated Reward, a method that mitigates temporal modeling degradation during fine-tuning. Extensive experiments, both qualitative and quantitative, validate the practicality and efficacy of using image reward models in InstructVideo, significantly enhancing the visual quality of generated videos without compromising generalization capabilities. Code and models will be made publicly available.