SMRABooth: Subject and Motion Representation Alignment for Customized Video Generation

TL;DR

SMRABooth introduces a novel method for customized video generation that aligns subject appearance and motion at the object level using self-supervised and optical flow encoders, improving control and consistency.

Contribution

The paper presents a new approach combining object-level subject and motion representations with sparse LoRA injection, enhancing personalized video generation.

Findings

Outperforms existing methods in subject appearance preservation.

Maintains consistent motion patterns across generated videos.

Effective in controllable text-to-video generation tasks.

Abstract

Customized video generation aims to produce videos that faithfully preserve the subject's appearance from reference images while maintaining temporally consistent motion from reference videos. Existing methods struggle to ensure both subject appearance similarity and motion pattern consistency due to the lack of object-level guidance for subject and motion. To address this, we propose SMRABooth, which leverages the self-supervised encoder and optical flow encoder to provide object-level subject and motion representations. These representations are aligned with the model during the LoRA fine-tuning process. Our approach is structured in three core stages: (1) We exploit subject representations via a self-supervised encoder to guide subject alignment, enabling the model to capture overall structure of subject and enhance high-level semantic consistency. (2) We utilize motion representations from an optical flow encoder to capture structurally coherent and object-level motion trajectories independent of appearance. (3) We propose a subject-motion association decoupling strategy that applies sparse LoRAs injection across both locations and timing, effectively reducing interference between subject and motion LoRAs. Extensive experiments show that SMRABooth excels in subject and motion customization, maintaining consistent subject appearance and motion patterns, proving its effectiveness in controllable text-to-video generation.