GenVideo: One-shot Target-image and Shape Aware Video Editing using T2I Diffusion Models

TL;DR

GenVideo introduces a novel video editing method using diffusion models that incorporates target images and shape awareness to improve control and consistency in edits involving objects of varying shapes and sizes.

Contribution

The paper presents a new approach that integrates target-image and shape awareness into diffusion-based video editing, enhancing control and temporal consistency over existing methods.

Findings

Effectively handles edits with objects of varying shapes and sizes.

Maintains temporal consistency during video editing.

Outperforms existing approaches in handling shape and size variations.

Abstract

Video editing methods based on diffusion models that rely solely on a text prompt for the edit are hindered by the limited expressive power of text prompts. Thus, incorporating a reference target image as a visual guide becomes desirable for precise control over edit. Also, most existing methods struggle to accurately edit a video when the shape and size of the object in the target image differ from the source object. To address these challenges, we propose "GenVideo" for editing videos leveraging target-image aware T2I models. Our approach handles edits with target objects of varying shapes and sizes while maintaining the temporal consistency of the edit using our novel target and shape aware InvEdit masks. Further, we propose a novel target-image aware latent noise correction strategy during inference to improve the temporal consistency of the edits. Experimental analyses indicate that GenVideo can effectively handle edits with objects of varying shapes, where existing approaches fail.