Edit as You See: Image-guided Video Editing via Masked Motion Modeling

TL;DR

This paper introduces IVEDiff, a novel image-guided video editing diffusion model that maintains temporal consistency and high quality in edited videos using masked motion modeling and optical flow guidance.

Contribution

The paper proposes a new image-guided video editing model with learnable motion modules and a masked motion modeling strategy, enhancing temporal consistency and editing accuracy.

Findings

Produces temporally smooth edited videos

Effectively handles various editing objects

Maintains high editing quality

Abstract

Recent advancements in diffusion models have significantly facilitated text-guided video editing. However, there is a relative scarcity of research on image-guided video editing, a method that empowers users to edit videos by merely indicating a target object in the initial frame and providing an RGB image as reference, without relying on the text prompts. In this paper, we propose a novel Image-guided Video Editing Diffusion model, termed IVEDiff for the image-guided video editing. IVEDiff is built on top of image editing models, and is equipped with learnable motion modules to maintain the temporal consistency of edited video. Inspired by self-supervised learning concepts, we introduce a masked motion modeling fine-tuning strategy that empowers the motion module's capabilities for capturing inter-frame motion dynamics, while preserving the capabilities for intra-frame semantic correlations modeling of the base image editing model. Moreover, an optical-flow-guided motion reference network is proposed to ensure the accurate propagation of information between edited video frames, alleviating the misleading effects of invalid information. We also construct a benchmark to facilitate further research. The comprehensive experiments demonstrate that our method is able to generate temporally smooth edited videos while robustly dealing with various editing objects with high quality.