DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan, Liu, Xintao Wang, Tien-Tsin Wong, Ying Shan

TL;DR
DynamiCrafter introduces a novel method for animating open-domain images by leveraging text-to-video diffusion models and image guidance, producing natural and convincing animated videos from static images.
Contribution
The paper presents a new approach that combines diffusion priors with image guidance to animate diverse images beyond traditional domain-specific methods.
Findings
Produces visually convincing animations
Achieves higher conformity to input images
Outperforms existing animation techniques
Abstract
Animating a still image offers an engaging visual experience. Traditional image animation techniques mainly focus on animating natural scenes with stochastic dynamics (e.g. clouds and fluid) or domain-specific motions (e.g. human hair or body motions), and thus limits their applicability to more general visual content. To overcome this limitation, we explore the synthesis of dynamic content for open-domain images, converting them into animated videos. The key idea is to utilize the motion prior of text-to-video diffusion models by incorporating the image into the generative process as guidance. Given an image, we first project it into a text-aligned rich context representation space using a query transformer, which facilitates the video model to digest the image content in a compatible fashion. However, some visual details still struggle to be preserved in the resultant videos. To…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
- 🤗Doubiiu/DynamiCraftermodel· ♡ 25♡ 25
- 🤗Doubiiu/DynamiCrafter_512model· ♡ 12♡ 12
- 🤗Doubiiu/DynamiCrafter_1024model· ♡ 78♡ 78
- 🤗Doubiiu/DynamiCrafter_512_Interpmodel· ♡ 42♡ 42
- 🤗pharaouk/DynamiCrafter_1024model
- 🤗pharaouk/DynamiCrafter_512model
- 🤗Kijai/DynamiCrafter_prunedmodel· ♡ 70♡ 70
- 🤗ReySajju742/VideoCraftermodel
- 🤗Jehj/DynamiCrafter_1024model
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Human Motion and Animation · Computer Graphics and Visualization Techniques
MethodsFocus · Diffusion
