Phantom: Subject-consistent video generation via cross-modal alignment

TL;DR

Phantom is a novel framework that achieves high-fidelity, subject-consistent video generation by aligning text and image prompts with video content, improving over existing methods especially in multi-subject scenarios.

Contribution

We introduce Phantom, a unified cross-modal alignment framework that enhances subject consistency in video generation from text and images, addressing content leakage and multi-subject confusion.

Findings

Outperforms state-of-the-art commercial solutions.

Achieves high-fidelity, subject-consistent videos.

Effectively handles multi-subject references.

Abstract

The continuous development of foundational models for video generation is evolving into various applications, with subject-consistent video generation still in the exploratory stage. We refer to this as Subject-to-Video, which extracts subject elements from reference images and generates subject-consistent videos following textual instructions. We believe that the essence of subject-to-video lies in balancing the dual-modal prompts of text and image, thereby deeply and simultaneously aligning both text and visual content. To this end, we propose Phantom, a unified video generation framework for both single- and multi-subject references. Building on existing text-to-video and image-to-video architectures, we redesign the joint text-image injection model and drive it to learn cross-modal alignment via text-image-video triplet data. The proposed method achieves high-fidelity subject-consistent video generation while addressing issues of image content leakage and multi-subject confusion. Evaluation results indicate that our method outperforms other state-of-the-art closed-source commercial solutions. In particular, we emphasize subject consistency in human generation, covering existing ID-preserving video generation while offering enhanced advantages.