Ada-VE: Training-Free Consistent Video Editing Using Adaptive Motion Prior

TL;DR

Ada-VE introduces an efficient adaptive motion-guided attention mechanism for video editing that enhances temporal coherence and visual quality without increasing computational costs, outperforming existing methods.

Contribution

The paper proposes a novel adaptive motion-guided cross-frame attention with KV-caching, enabling more frames to be edited efficiently and improving temporal consistency in video synthesis.

Findings

Achieves a threefold increase in keyframe processing within the same computational budget.

Improves temporal coherence and visual quality over state-of-the-art methods.

Reduces redundant computations using optical flow-guided attention and KV-caching.

Abstract

Video-to-video synthesis poses significant challenges in maintaining character consistency, smooth temporal transitions, and preserving visual quality during fast motion. While recent fully cross-frame self-attention mechanisms have improved character consistency across multiple frames, they come with high computational costs and often include redundant operations, especially for videos with higher frame rates. To address these inefficiencies, we propose an adaptive motion-guided cross-frame attention mechanism that selectively reduces redundant computations. This enables a greater number of cross-frame attentions over more frames within the same computational budget, thereby enhancing both video quality and temporal coherence. Our method leverages optical flow to focus on moving regions while sparsely attending to stationary areas, allowing for the joint editing of more frames without increasing computational demands. Traditional frame interpolation techniques struggle with motion blur and flickering in intermediate frames, which compromises visual fidelity. To mitigate this, we introduce KV-caching for jointly edited frames, reusing keys and values across intermediate frames to preserve visual quality and maintain temporal consistency throughout the video. With our adaptive cross-frame self-attention approach, we achieve a threefold increase in the number of keyframes processed compared to existing methods, all within the same computational budget as fully cross-frame attention baselines. This results in significant improvements in prediction accuracy and temporal consistency, outperforming state-of-the-art approaches. Code will be made publicly available at https://github.com/tanvir-utexas/AdaVE/tree/main