Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Models

TL;DR

This paper introduces a method for one-shot motion customization in text-to-video diffusion models, enabling the transfer of motion from a single reference video to new subjects and scenes with spatial and temporal variations.

Contribution

It proposes a novel approach using LoRA on temporal attention layers and appearance absorbers to disentangle appearance and motion, facilitating flexible video customization.

Findings

Effective motion transfer from a single reference video

Enables various downstream video editing tasks

Plug-and-play inference for easy extension

Abstract

Image customization has been extensively studied in text-to-image (T2I) diffusion models, leading to impressive outcomes and applications. With the emergence of text-to-video (T2V) diffusion models, its temporal counterpart, motion customization, has not yet been well investigated. To address the challenge of one-shot video motion customization, we propose Customize-A-Video that models the motion from a single reference video and adapts it to new subjects and scenes with both spatial and temporal varieties. It leverages low-rank adaptation (LoRA) on temporal attention layers to tailor the pre-trained T2V diffusion model for specific motion modeling. To disentangle the spatial and temporal information during training, we introduce a novel concept of appearance absorbers that detach the original appearance from the reference video prior to motion learning. The proposed modules are trained in a staged pipeline and inferred in a plug-and-play fashion, enabling easy extensions to various downstream tasks such as custom video generation and editing, video appearance customization and multiple motion combination. Our project page can be found at https://customize-a-video.github.io.