Kaleido: Open-Sourced Multi-Subject Reference Video Generation Model

TL;DR

Kaleido is a novel multi-subject video generation framework that improves consistency and fidelity by using a new data pipeline and Reference Rotary Positional Encoding, advancing the state-of-the-art in subject-to-video synthesis.

Contribution

The paper introduces a new data construction pipeline and R-RoPE mechanism, significantly enhancing multi-subject consistency and fidelity in reference video generation.

Findings

Outperforms previous methods in multiple benchmarks

Achieves higher subject consistency and background disentanglement

Demonstrates improved generalization across diverse subjects

Abstract

We present Kaleido, a subject-to-video~(S2V) generation framework, which aims to synthesize subject-consistent videos conditioned on multiple reference images of target subjects. Despite recent progress in S2V generation models, existing approaches remain inadequate at maintaining multi-subject consistency and at handling background disentanglement, often resulting in lower reference fidelity and semantic drift under multi-image conditioning. These shortcomings can be attributed to several factors. Primarily, the training dataset suffers from a lack of diversity and high-quality samples, as well as cross-paired data, i.e., paired samples whose components originate from different instances. In addition, the current mechanism for integrating multiple reference images is suboptimal, potentially resulting in the confusion of multiple subjects. To overcome these limitations, we propose a dedicated data construction pipeline, incorporating low-quality sample filtering and diverse data synthesis, to produce consistency-preserving training data. Moreover, we introduce Reference Rotary Positional Encoding (R-RoPE) to process reference images, enabling stable and precise multi-image integration. Extensive experiments across numerous benchmarks demonstrate that Kaleido significantly outperforms previous methods in consistency, fidelity, and generalization, marking an advance in S2V generation.