DreamMover: Leveraging the Prior of Diffusion Models for Image Interpolation with Large Motion

TL;DR

DreamMover leverages diffusion models to generate semantically consistent intermediate images for large motion scenarios, addressing limitations of existing methods by reasoning about semantic correspondence and fusing multi-level information.

Contribution

We introduce DreamMover, a novel diffusion-based framework for image interpolation that handles large motions and semantic inconsistencies, along with a new benchmark dataset InterpBench.

Findings

Outperforms existing methods in semantic consistency

Effectively handles large motion interpolation

Demonstrates robustness on InterpBench dataset

Abstract

We study the problem of generating intermediate images from image pairs with large motion while maintaining semantic consistency. Due to the large motion, the intermediate semantic information may be absent in input images. Existing methods either limit to small motion or focus on topologically similar objects, leading to artifacts and inconsistency in the interpolation results. To overcome this challenge, we delve into pre-trained image diffusion models for their capabilities in semantic cognition and representations, ensuring consistent expression of the absent intermediate semantic representations with the input. To this end, we propose DreamMover, a novel image interpolation framework with three main components: 1) A natural flow estimator based on the diffusion model that can implicitly reason about the semantic correspondence between two images. 2) To avoid the loss of detailed information during fusion, our key insight is to fuse information in two parts, high-level space and low-level space. 3) To enhance the consistency between the generated images and input, we propose the self-attention concatenation and replacement approach. Lastly, we present a challenging benchmark dataset InterpBench to evaluate the semantic consistency of generated results. Extensive experiments demonstrate the effectiveness of our method. Our project is available at https://dreamm0ver.github.io .