TV-LiVE: Training-Free, Text-Guided Video Editing via Layer Informed Vitality Exploitation

TL;DR

TV-LiVE introduces a training-free, text-guided video editing method that exploits layer vitality in diffusion models to enable complex edits like object addition and non-rigid transformations.

Contribution

The paper identifies vital layers in diffusion models related to Rotary Position Embeddings and leverages them for effective, training-free video editing guided by text prompts.

Findings

Outperforms existing methods in object addition and non-rigid editing

Effectively identifies mask regions for new objects using prominent layers

Enables complex video edits without additional training or fine-tuning

Abstract

Video editing has garnered increasing attention alongside the rapid progress of diffusion-based video generation models. As part of these advancements, there is a growing demand for more accessible and controllable forms of video editing, such as prompt-based editing. Previous studies have primarily focused on tasks such as style transfer, background replacement, object substitution, and attribute modification, while maintaining the content structure of the source video. However, more complex tasks, including the addition of novel objects and nonrigid transformations, remain relatively unexplored. In this paper, we present TV-LiVE, a Training-free and text-guided Video editing framework via Layerinformed Vitality Exploitation. We empirically identify vital layers within the video generation model that significantly influence the quality of generated outputs. Notably, these layers are closely associated with Rotary Position Embeddings (RoPE). Based on this observation, our method enables both object addition and non-rigid video editing by selectively injecting key and value features from the source model into the corresponding layers of the target model guided by the layer vitality. For object addition, we further identify prominent layers to extract the mask regions corresponding to the newly added target prompt. We found that the extracted masks from the prominent layers faithfully indicate the region to be edited. Experimental results demonstrate that TV-LiVE outperforms existing approaches for both object addition and non-rigid video editing. Project Page: https://emjay73.github.io/TV_LiVE/