Pathways on the Image Manifold: Image Editing via Video Generation

TL;DR

This paper introduces a novel image editing method that leverages video generation models to produce smooth, consistent edits by traversing the image manifold, significantly improving accuracy and fidelity over existing diffusion-based techniques.

Contribution

It proposes reformulating image editing as a temporal process using pretrained video models, enabling continuous and faithful edits that preserve key image elements.

Findings

Achieves state-of-the-art results on text-based image editing

Improves edit accuracy and image fidelity

Creates smooth transitions from original to edited images

Abstract

Recent advances in image editing, driven by image diffusion models, have shown remarkable progress. However, significant challenges remain, as these models often struggle to follow complex edit instructions accurately and frequently compromise fidelity by altering key elements of the original image. Simultaneously, video generation has made remarkable strides, with models that effectively function as consistent and continuous world simulators. In this paper, we propose merging these two fields by utilizing image-to-video models for image editing. We reformulate image editing as a temporal process, using pretrained video models to create smooth transitions from the original image to the desired edit. This approach traverses the image manifold continuously, ensuring consistent edits while preserving the original image's key aspects. Our approach achieves state-of-the-art results on text-based image editing, demonstrating significant improvements in both edit accuracy and image preservation. Visit our project page at https://rotsteinnoam.github.io/Frame2Frame.