Magic-Me: Identity-Specific Video Customized Diffusion

TL;DR

Magic-Me introduces a novel framework for identity-specific video generation that maintains consistent identity features across frames using innovative initialization, identity disentanglement, and face reinforcement techniques.

Contribution

The paper presents a new Video Custom Diffusion framework with three key components for stable, identity-preserving video synthesis from limited reference images.

Findings

VCD produces stable videos with improved identity preservation.

The method effectively disentangles identity from background.

VCD works well with publicly available personalized text-to-image models.

Abstract

Creating content with specified identities (ID) has attracted significant interest in the field of generative models. In the field of text-to-image generation (T2I), subject-driven creation has achieved great progress with the identity controlled via reference images. However, its extension to video generation is not well explored. In this work, we propose a simple yet effective subject identity controllable video generation framework, termed Video Custom Diffusion (VCD). With a specified identity defined by a few images, VCD reinforces the identity characteristics and injects frame-wise correlation at the initialization stage for stable video outputs. To achieve this, we propose three novel components that are essential for high-quality identity preservation and stable video generation: 1) a noise initialization method with 3D Gaussian Noise Prior for better inter-frame stability; 2) an ID module based on extended Textual Inversion trained with the cropped identity to disentangle the ID information from the background 3) Face VCD and Tiled VCD modules to reinforce faces and upscale the video to higher resolution while preserving the identity's features. We conducted extensive experiments to verify that VCD is able to generate stable videos with better ID over the baselines. Besides, with the transferability of the encoded identity in the ID module, VCD is also working well with personalized text-to-image models available publicly. The codes are available at https://github.com/Zhen-Dong/Magic-Me.