Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning

TL;DR

Control-A-Video introduces a controllable text-to-video diffusion model that incorporates content and motion priors, along with reward feedback learning, to generate high-quality, motion-consistent videos guided by text prompts and control maps.

Contribution

The paper presents novel strategies for integrating content and motion priors into diffusion-based video generation and introduces a reward feedback learning algorithm for improved quality and consistency.

Findings

Produces higher-quality videos than existing methods.

Achieves better motion consistency and relevance.

Demonstrates effectiveness of reward feedback learning.

Abstract

Recent advances in text-to-image (T2I) diffusion models have enabled impressive image generation capabilities guided by text prompts. However, extending these techniques to video generation remains challenging, with existing text-to-video (T2V) methods often struggling to produce high-quality and motion-consistent videos. In this work, we introduce Control-A-Video, a controllable T2V diffusion model that can generate videos conditioned on text prompts and reference control maps like edge and depth maps. To tackle video quality and motion consistency issues, we propose novel strategies to incorporate content prior and motion prior into the diffusion-based generation process. Specifically, we employ a first-frame condition scheme to transfer video generation from the image domain. Additionally, we introduce residual-based and optical flow-based noise initialization to infuse motion priors from reference videos, promoting relevance among frame latents for reduced flickering. Furthermore, we present a Spatio-Temporal Reward Feedback Learning (ST-ReFL) algorithm that optimizes the video diffusion model using multiple reward models for video quality and motion consistency, leading to superior outputs. Comprehensive experiments demonstrate that our framework generates higher-quality, more consistent videos compared to existing state-of-the-art methods in controllable text-to-video generation