DualReal: Adaptive Joint Training for Lossless Identity-Motion Fusion in Video Customization

TL;DR

DualReal introduces an adaptive joint training framework that effectively fuses identity and motion in text-to-video generation, overcoming conflicts of previous isolated methods and significantly improving consistency and quality.

Contribution

The paper proposes DualReal, a novel framework with adaptive joint training and stage-guided control to enhance identity-motion fusion in video synthesis.

Findings

Improves CLIP-I by 21.7% and DINO-I by 31.8% on average.

Achieves top performance on nearly all motion metrics.

Constructs a comprehensive evaluation benchmark.

Abstract

Customized text-to-video generation with pre-trained large-scale models has recently garnered significant attention by focusing on identity and motion consistency. Existing works typically follow the isolated customized paradigm, where the subject identity or motion dynamics are customized exclusively. However, this paradigm completely ignores the intrinsic mutual constraints and synergistic interdependencies between identity and motion, resulting in identity-motion conflicts throughout the generation process that systematically degrade. To address this, we introduce DualReal, a novel framework that employs adaptive joint training to construct interdependencies between dimensions collaboratively. Specifically, DualReal is composed of two units: (1) Dual-aware Adaptation dynamically switches the training step (i.e., identity or motion), learns the current information guided by the frozen dimension prior, and employs a regularization strategy to avoid knowledge leakage; (2) StageBlender Controller leverages the denoising stages and Diffusion Transformer depths to guide different dimensions with adaptive granularity, avoiding conflicts at various stages and ultimately achieving lossless fusion of identity and motion patterns. We constructed a more comprehensive evaluation benchmark than existing methods. The experimental results show that DualReal improves CLIP-I and DINO-I metrics by 21.7% and 31.8% on average, and achieves top performance on nearly all motion metrics. Page: https://wenc-k.github.io/dualreal-customization