Training-Free Video Editing via Optical Flow-Enhanced Score Distillation

TL;DR

This paper introduces a training-free video editing method using score distillation guided by pre-trained text-to-video models, which improves content preservation, temporal continuity, and global consistency over existing techniques.

Contribution

It proposes a novel score distillation framework with iterative optimization and auxiliary losses to enhance unedited region preservation and temporal coherence in training-free video editing.

Findings

Effective preservation of unedited regions

Improved temporal continuity and content consistency

Outperforms state-of-the-art methods in multiple metrics

Abstract

The rapid advancement in visual generation, particularly the emergence of pre-trained text-to-image and text-to-video models, has catalyzed growing interest in training-free video editing research. Mirroring training-free image editing techniques, current approaches preserve original video information through video input inversion and manipulating intermediate features and attention during the inference process to achieve content editing. Although they have demonstrated promising results, the lossy nature of the inversion process poses significant challenges in maintaining unedited regions of the video. Furthermore, feature and attention manipulation during inference can lead to unintended over-editing and face challenges in both local temporal continuity and global content consistency. To address these challenges, this study proposes a score distillation paradigm based on pre-trained text-to-video models, where the original video is iteratively optimized through multiple steps guided by editing gradients provided by score distillation to ultimately obtain the target video. The iterative optimization starting from the original video, combined with content preservation loss, ensures the maintenance of unedited regions in the original video and suppresses over-editing. To further guarantee video content consistency and temporal continuity, we additionally introduce a global consistency auxiliary loss and optical flow prediction-based local editing gradient smoothing. Experiments demonstrate that these strategies effectively address the aforementioned challenges, achieving comparable or superior performance across multiple dimensions including preservation of unedited regions, local temporal continuity, and global content consistency of editing results, compared to state-of-the-art methods.