AdaFlow: Efficient Long Video Editing via Adaptive Attention Slimming And Keyframe Selection

TL;DR

AdaFlow introduces an adaptive, training-free method for efficient long video editing by selectively slimming attention and choosing keyframes, enabling editing of videos over ten times longer than previous methods.

Contribution

The paper proposes AdaFlow, a novel approach that adaptively reduces attention complexity and selects keyframes, significantly improving long video editing efficiency and quality without additional training.

Findings

AdaFlow can edit over 1,000 frames in one inference on a single GPU.

It achieves about ten times longer video editing than previous methods like TokenFlow.

The approach maintains high-quality editing with adaptive attention and keyframe selection.

Abstract

Despite great progress, text-driven long video editing is still notoriously challenging mainly due to excessive memory overhead. Although recent efforts have simplified this task into a two-step process of keyframe translation and interpolation generation, the token-wise keyframe translation still plagues the upper limit of video length. In this paper, we propose a novel and training-free approach towards efficient and effective long video editing, termed AdaFlow. We first reveal that not all tokens of video frames hold equal importance for keyframe translation, based on which we propose an Adaptive Attention Slimming scheme for AdaFlow to squeeze the $KV$ sequence, thus increasing the number of keyframes for translations by an order of magnitude. In addition, an Adaptive Keyframe Selection scheme is also equipped to select the representative frames for joint editing, further improving generation quality. With these innovative designs, AdaFlow achieves high-quality long video editing of minutes in one inference, i.e., more than 1$k$ frames on one A800 GPU, which is about ten times longer than the compared methods, e.g., TokenFlow. To validate AdaFlow, we also build a new benchmark for long video editing with high-quality annotations, termed LongV-EVAL. Our code is released at: https://github.com/jidantang55/AdaFlow.