ID-Crafter: VLM-Grounded Online RL for Compositional Multi-Subject Video Generation

TL;DR

ID-Crafter introduces a novel framework for multi-subject video generation that combines hierarchical attention, vision-language understanding, and reinforcement learning to improve identity preservation and semantic coherence in generated videos.

Contribution

The paper presents a new multi-subject video generation method integrating hierarchical attention, pretrained vision-language models, and online reinforcement learning, along with a new dataset for training and evaluation.

Findings

Achieves state-of-the-art results in identity preservation and semantic coherence.

Demonstrates superior temporal consistency and video quality.

Outperforms existing methods on multiple benchmarks.

Abstract

Significant progress has been achieved in high-fidelity video synthesis, yet current paradigms often fall short in effectively integrating identity information from multiple subjects. This leads to semantic conflicts and suboptimal performance in preserving identities and interactions, limiting controllability and applicability. To tackle this issue, we introduce ID-Crafter, a framework for multi-subject video generation that achieves superior identity preservation and semantic coherence. ID-Crafter integrates three key components: (i) a hierarchical identity-preserving attention mechanism that progressively aggregates features at intra-subject, inter-subject, and cross-modal levels; (ii) a semantic understanding module powered by a pretrained Vision-Language Model (VLM) to provide fine-grained guidance and capture complex inter-subject relationships; and (iii) an online reinforcement learning phase to further refine the model for critical concepts. Furthermore, we construct a new dataset to facilitate robust training and evaluation. Extensive experiments demonstrate that ID-Crafter establishes new state-of-the-art performance on multi-subject video generation benchmarks, excelling in identity preservation, temporal consistency, and overall video quality. Project page: https://angericky.github.io/ID-Crafter