Controllable Longer Image Animation with Diffusion Models

TL;DR

This paper presents a novel diffusion-based method for controllable, long-duration image animation that allows precise motion control and maintains content consistency over 100 frames, surpassing existing short-video limitations.

Contribution

It introduces a new long-duration video generation approach with noise rescheduling, enabling controllable and coherent animations from static images.

Findings

Outperforms 10 baseline methods in experiments

Enables creation of videos over 100 frames with content consistency

Provides precise control over motion direction and speed

Abstract

Generating realistic animated videos from static images is an important area of research in computer vision. Methods based on physical simulation and motion prediction have achieved notable advances, but they are often limited to specific object textures and motion trajectories, failing to exhibit highly complex environments and physical dynamics. In this paper, we introduce an open-domain controllable image animation method using motion priors with video diffusion models. Our method achieves precise control over the direction and speed of motion in the movable region by extracting the motion field information from videos and learning moving trajectories and strengths. Current pretrained video generation models are typically limited to producing very short videos, typically less than 30 frames. In contrast, we propose an efficient long-duration video generation method based on noise reschedule specifically tailored for image animation tasks, facilitating the creation of videos over 100 frames in length while maintaining consistency in content scenery and motion coordination. Specifically, we decompose the denoise process into two distinct phases: the shaping of scene contours and the refining of motion details. Then we reschedule the noise to control the generated frame sequences maintaining long-distance noise correlation. We conducted extensive experiments with 10 baselines, encompassing both commercial tools and academic methodologies, which demonstrate the superiority of our method. Our project page: https://wangqiang9.github.io/Controllable.github.io/