DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors

TL;DR

DynamiCrafter introduces a novel method for animating open-domain images by leveraging text-to-video diffusion models and image guidance, producing natural and convincing animated videos from static images.

Contribution

The paper presents a new approach that combines diffusion priors with image guidance to animate diverse images beyond traditional domain-specific methods.

Findings

Produces visually convincing animations

Achieves higher conformity to input images

Outperforms existing animation techniques

Abstract

Animating a still image offers an engaging visual experience. Traditional image animation techniques mainly focus on animating natural scenes with stochastic dynamics (e.g. clouds and fluid) or domain-specific motions (e.g. human hair or body motions), and thus limits their applicability to more general visual content. To overcome this limitation, we explore the synthesis of dynamic content for open-domain images, converting them into animated videos. The key idea is to utilize the motion prior of text-to-video diffusion models by incorporating the image into the generative process as guidance. Given an image, we first project it into a text-aligned rich context representation space using a query transformer, which facilitates the video model to digest the image content in a compatible fashion. However, some visual details still struggle to be preserved in the resultant videos. To supplement with more precise image information, we further feed the full image to the diffusion model by concatenating it with the initial noises. Experimental results show that our proposed method can produce visually convincing and more logical & natural motions, as well as higher conformity to the input image. Comparative evaluation demonstrates the notable superiority of our approach over existing competitors.