MotionClone: Training-Free Motion Cloning for Controllable Video Generation

TL;DR

MotionClone introduces a training-free method for controllable video generation that transfers motion from reference videos using sparse temporal attention, enabling flexible and efficient motion cloning without model retraining.

Contribution

It proposes a novel training-free framework leveraging temporal attention for motion cloning, eliminating the need for model training or fine-tuning.

Findings

Effective in transferring both global and local motions

Outperforms existing methods in motion fidelity and temporal consistency

Enables versatile motion-controlled video generation from various inputs

Abstract

Motion-based controllable video generation offers the potential for creating captivating visual content. Existing methods typically necessitate model training to encode particular motion cues or incorporate fine-tuning to inject certain motion patterns, resulting in limited flexibility and generalization. In this work, we propose MotionClone, a training-free framework that enables motion cloning from reference videos to versatile motion-controlled video generation, including text-to-video and image-to-video. Based on the observation that the dominant components in temporal-attention maps drive motion synthesis, while the rest mainly capture noisy or very subtle motions, MotionClone utilizes sparse temporal attention weights as motion representations for motion guidance, facilitating diverse motion transfer across varying scenarios. Meanwhile, MotionClone allows for the direct extraction of motion representation through a single denoising step, bypassing the cumbersome inversion processes and thus promoting both efficiency and flexibility. Extensive experiments demonstrate that MotionClone exhibits proficiency in both global camera motion and local object motion, with notable superiority in terms of motion fidelity, textual alignment, and temporal consistency.