Shape-aware Text-driven Layered Video Editing

TL;DR

This paper introduces a shape-aware, text-driven layered video editing method that effectively handles shape changes and ensures temporal consistency, advancing beyond existing texture-focused approaches.

Contribution

It proposes a novel approach combining deformation propagation and diffusion guidance to enable shape-aware, consistent video editing driven by text prompts.

Findings

Achieves shape-aware, temporally consistent video edits

Outperforms state-of-the-art methods in qualitative comparisons

Effectively handles shape changes and unseen regions

Abstract

Temporal consistency is essential for video editing applications. Existing work on layered representation of videos allows propagating edits consistently to each frame. These methods, however, can only edit object appearance rather than object shape changes due to the limitation of using a fixed UV mapping field for texture atlas. We present a shape-aware, text-driven video editing method to tackle this challenge. To handle shape changes in video editing, we first propagate the deformation field between the input and edited keyframe to all frames. We then leverage a pre-trained text-conditioned diffusion model as guidance for refining shape distortion and completing unseen regions. The experimental results demonstrate that our method can achieve shape-aware consistent video editing and compare favorably with the state-of-the-art.