Zero-Shot Video Translation and Editing with Frame Spatial-Temporal Correspondence

TL;DR

FRESCO is a novel framework that improves zero-shot video translation and editing by integrating intra- and inter-frame correspondence, ensuring high spatial-temporal consistency and visual coherence in manipulated videos.

Contribution

The paper introduces FRESCO, a new method that explicitly optimizes features for zero-shot video translation and editing, surpassing existing attention-based approaches in temporal consistency.

Findings

FRESCO achieves superior spatial-temporal consistency in manipulated videos.

The method outperforms current zero-shot techniques in video quality and coherence.

Experiments confirm the effectiveness of FRESCO on video translation and editing tasks.

Abstract

The remarkable success in text-to-image diffusion models has motivated extensive investigation of their potential for video applications. Zero-shot techniques aim to adapt image diffusion models for videos without requiring further model training. Recent methods largely emphasize integrating inter-frame correspondence into attention mechanisms. However, the soft constraint applied to identify the valid features to attend is insufficient, which could lead to temporal inconsistency. In this paper, we present FRESCO, which integrates intra-frame correspondence with inter-frame correspondence to formulate a more robust spatial-temporal constraint. This enhancement ensures a consistent transformation of semantically similar content between frames. Our method goes beyond attention guidance to explicitly optimize features, achieving high spatial-temporal consistency with the input video, significantly enhancing the visual coherence of manipulated videos. We verify FRESCO adaptations on two zero-shot tasks of video-to-video translation and text-guided video editing. Comprehensive experiments demonstrate the effectiveness of our framework in generating high-quality, coherent videos, highlighting a significant advance over current zero-shot methods.