MotionMatcher: Motion Customization of Text-to-Video Diffusion Models via Motion Feature Matching

TL;DR

MotionMatcher is a novel framework that enables precise motion customization in text-to-video diffusion models by matching high-level motion features, overcoming limitations of previous pixel-level fine-tuning methods.

Contribution

We introduce MotionMatcher, a feature-level fine-tuning approach for T2V models that improves motion accuracy and reduces content leakage compared to existing methods.

Findings

Achieves state-of-the-art motion customization performance.

Effectively captures complex motion with high fidelity.

Reduces content leakage from reference videos.

Abstract

Text-to-video (T2V) diffusion models have shown promising capabilities in synthesizing realistic videos from input text prompts. However, the input text description alone provides limited control over the precise objects movements and camera framing. In this work, we tackle the motion customization problem, where a reference video is provided as motion guidance. While most existing methods choose to fine-tune pre-trained diffusion models to reconstruct the frame differences of the reference video, we observe that such strategy suffer from content leakage from the reference video, and they cannot capture complex motion accurately. To address this issue, we propose MotionMatcher, a motion customization framework that fine-tunes the pre-trained T2V diffusion model at the feature level. Instead of using pixel-level objectives, MotionMatcher compares high-level, spatio-temporal motion features to fine-tune diffusion models, ensuring precise motion learning. For the sake of memory efficiency and accessibility, we utilize a pre-trained T2V diffusion model, which contains considerable prior knowledge about video motion, to compute these motion features. In our experiments, we demonstrate state-of-the-art motion customization performances, validating the design of our framework.