Temporally Consistent Object Editing in Videos using Extended Attention

TL;DR

This paper introduces a novel video editing method that uses extended attention modules in a pre-trained inpainting diffusion model to achieve temporally consistent edits across frames, outperforming existing techniques.

Contribution

The work presents a new approach to video editing by redesigning the diffusion model's attention mechanism to ensure frame-level temporal consistency.

Findings

Achieves superior accuracy in object retargeting, replacement, and removal tasks.

Ensures consistent editing across all video frames regardless of mask shape and position.

Outperforms state-of-the-art methods in qualitative comparisons.

Abstract

Image generation and editing have seen a great deal of advancements with the rise of large-scale diffusion models that allow user control of different modalities such as text, mask, depth maps, etc. However, controlled editing of videos still lags behind. Prior work in this area has focused on using 2D diffusion models to globally change the style of an existing video. On the other hand, in many practical applications, editing localized parts of the video is critical. In this work, we propose a method to edit videos using a pre-trained inpainting image diffusion model. We systematically redesign the forward path of the model by replacing the self-attention modules with an extended version of attention modules that creates frame-level dependencies. In this way, we ensure that the edited information will be consistent across all the video frames no matter what the shape and position of the masked area is. We qualitatively compare our results with state-of-the-art in terms of accuracy on several video editing tasks like object retargeting, object replacement, and object removal tasks. Simulations demonstrate the superior performance of the proposed strategy.