EchoShot: Multi-Shot Portrait Video Generation

TL;DR

EchoShot is a scalable multi-shot portrait video generation framework that ensures identity consistency and attribute control, built on a novel video diffusion transformer and a new large-scale dataset.

Contribution

It introduces shot-aware position embeddings for multi-shot modeling and constructs PortraitGala, a high-fidelity dataset for training and evaluation.

Findings

Achieves superior identity consistency in multi-shot videos

Enables attribute-level controllability in portrait video synthesis

Supports reference image-based personalized and long video generation

Abstract

Video diffusion models substantially boost the productivity of artistic workflows with high-quality portrait video generative capacity. However, prevailing pipelines are primarily constrained to single-shot creation, while real-world applications urge for multiple shots with identity consistency and flexible content controllability. In this work, we propose EchoShot, a native and scalable multi-shot framework for portrait customization built upon a foundation video diffusion model. To start with, we propose shot-aware position embedding mechanisms within video diffusion transformer architecture to model inter-shot variations and establish intricate correspondence between multi-shot visual content and their textual descriptions. This simple yet effective design enables direct training on multi-shot video data without introducing additional computational overhead. To facilitate model training within multi-shot scenario, we construct PortraitGala, a large-scale and high-fidelity human-centric video dataset featuring cross-shot identity consistency and fine-grained captions such as facial attributes, outfits, and dynamic motions. To further enhance applicability, we extend EchoShot to perform reference image-based personalized multi-shot generation and long video synthesis with infinite shot counts. Extensive evaluations demonstrate that EchoShot achieves superior identity consistency as well as attribute-level controllability in multi-shot portrait video generation. Notably, the proposed framework demonstrates potential as a foundational paradigm for general multi-shot video modeling.