AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning

TL;DR

AnimateDiff introduces a versatile framework that enables the animation of personalized text-to-image diffusion models without needing model-specific tuning, by learning transferable motion priors from videos.

Contribution

The paper presents a plug-and-play motion module and MotionLoRA fine-tuning technique that facilitate animation of personalized T2I models with minimal additional training.

Findings

Generates smooth, high-quality animations from personalized T2I models.

The motion module is transferable across models from the same base.

MotionLoRA efficiently adapts to new motion patterns with low cost.

Abstract

With the advance of text-to-image (T2I) diffusion models (e.g., Stable Diffusion) and corresponding personalization techniques such as DreamBooth and LoRA, everyone can manifest their imagination into high-quality images at an affordable cost. However, adding motion dynamics to existing high-quality personalized T2Is and enabling them to generate animations remains an open challenge. In this paper, we present AnimateDiff, a practical framework for animating personalized T2I models without requiring model-specific tuning. At the core of our framework is a plug-and-play motion module that can be trained once and seamlessly integrated into any personalized T2Is originating from the same base T2I. Through our proposed training strategy, the motion module effectively learns transferable motion priors from real-world videos. Once trained, the motion module can be inserted into a personalized T2I model to form a personalized animation generator. We further propose MotionLoRA, a lightweight fine-tuning technique for AnimateDiff that enables a pre-trained motion module to adapt to new motion patterns, such as different shot types, at a low training and data collection cost. We evaluate AnimateDiff and MotionLoRA on several public representative personalized T2I models collected from the community. The results demonstrate that our approaches help these models generate temporally smooth animation clips while preserving the visual quality and motion diversity. Codes and pre-trained weights are available at https://github.com/guoyww/AnimateDiff.