Consistent Video Editing as Flow-Driven Image-to-Video Generation

TL;DR

This paper introduces FlowV2V, a flow-driven image-to-video generation method that improves complex motion modeling and temporal consistency in video editing, especially for non-rigid objects, outperforming existing methods.

Contribution

FlowV2V redefines video editing as flow-driven I2V generation, effectively modeling complex motions and ensuring temporal consistency through flow alignment and first-frame editing.

Findings

Achieves 13.67% improvement on DOVER metric

Reduces warping error by 50.66%

Outperforms state-of-the-art methods in temporal consistency

Abstract

With the prosper of video diffusion models, down-stream applications like video editing have been significantly promoted without consuming much computational cost. One particular challenge in this task lies at the motion transfer process from the source video to the edited one, where it requires the consideration of the shape deformation in between, meanwhile maintaining the temporal consistency in the generated video sequence. However, existing methods fail to model complicated motion patterns for video editing, and are fundamentally limited to object replacement, where tasks with non-rigid object motions like multi-object and portrait editing are largely neglected. In this paper, we observe that optical flows offer a promising alternative in complex motion modeling, and present FlowV2V to re-investigate video editing as a task of flow-driven Image-to-Video (I2V) generation. Specifically, FlowV2V decomposes the entire pipeline into first-frame editing and conditional I2V generation, and simulates pseudo flow sequence that aligns with the deformed shape, thus ensuring the consistency during editing. Experimental results on DAVIS-EDIT with improvements of 13.67% and 50.66% on DOVER and warping error illustrate the superior temporal consistency and sample quality of FlowV2V compared to existing state-of-the-art ones. Furthermore, we conduct comprehensive ablation studies to analyze the internal functionalities of the first-frame paradigm and flow alignment in the proposed method.