AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks

TL;DR

AnyV2V introduces a tuning-free, versatile framework for video editing that leverages existing image editing and image-to-video models, enabling high-quality, consistent edits across various tasks without extensive fine-tuning.

Contribution

It presents a novel, tuning-free approach that simplifies video editing into two steps, supporting diverse editing tasks and any video length, outperforming prior methods in human evaluations.

Findings

Achieved CLIP-scores comparable to baseline methods.

Significantly outperformed baselines in human evaluations.

Produced high-quality, visually consistent video edits.

Abstract

In the dynamic field of digital content creation using generative models, state-of-the-art video editing models still do not offer the level of quality and control that users desire. Previous works on video editing either extended from image-based generative models in a zero-shot manner or necessitated extensive fine-tuning, which can hinder the production of fluid video edits. Furthermore, these methods frequently rely on textual input as the editing guidance, leading to ambiguities and limiting the types of edits they can perform. Recognizing these challenges, we introduce AnyV2V, a novel tuning-free paradigm designed to simplify video editing into two primary steps: (1) employing an off-the-shelf image editing model to modify the first frame, (2) utilizing an existing image-to-video generation model to generate the edited video through temporal feature injection. AnyV2V can leverage any existing image editing tools to support an extensive array of video editing tasks, including prompt-based editing, reference-based style transfer, subject-driven editing, and identity manipulation, which were unattainable by previous methods. AnyV2V can also support any video length. Our evaluation shows that AnyV2V achieved CLIP-scores comparable to other baseline methods. Furthermore, AnyV2V significantly outperformed these baselines in human evaluations, demonstrating notable improvements in visual consistency with the source video while producing high-quality edits across all editing tasks.