ControlVideo: Conditional Control for One-shot Text-driven Video Editing and Beyond
Min Zhao, Rongzhen Wang, Fan Bao, Chongxuan Li, Jun Zhu

TL;DR
ControlVideo introduces a novel method for text-driven video editing that enhances fidelity and temporal consistency by leveraging conditional diffusion models, fine-tuning techniques, and extending to long videos with high frame counts.
Contribution
The paper presents ControlVideo, a new framework that improves one-shot text-driven video editing by incorporating additional conditions, fine-tuning strategies, and a method for long video editing with high temporal consistency.
Findings
Outperforms baselines in fidelity and temporal consistency.
Capable of editing videos with over 140 frames, significantly more than previous methods.
Effectively extends to long videos using a fused approach.
Abstract
This paper presents \emph{ControlVideo} for text-driven video editing -- generating a video that aligns with a given text while preserving the structure of the source video. Building on a pre-trained text-to-image diffusion model, ControlVideo enhances the fidelity and temporal consistency by incorporating additional conditions (such as edge maps), and fine-tuning the key-frame and temporal attention on the source video-text pair via an in-depth exploration of the design space. Extensive experimental results demonstrate that ControlVideo outperforms various competitive baselines by delivering videos that exhibit high fidelity w.r.t. the source content, and temporal consistency, all while aligning with the text. By incorporating Low-rank adaptation layers into the model before training, ControlVideo is further empowered to generate videos that align seamlessly with reference images. More…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsVideo Analysis and Summarization · Generative Adversarial Networks and Image Synthesis
MethodsDiffusion · ALIGN
