OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditions

TL;DR

OmniVCus introduces a novel diffusion Transformer framework for multi-subject, subject-driven video customization using multimodal control signals, with a new data construction pipeline enabling instructive editing.

Contribution

The paper presents a new data construction pipeline and a diffusion Transformer framework with innovative embedding mechanisms for multi-subject video customization.

Findings

Outperforms state-of-the-art methods in quantitative evaluations.

Effectively incorporates multimodal control signals for video editing.

Enables multi-subject inference with enhanced embedding techniques.

Abstract

Existing feedforward subject-driven video customization methods mainly study single-subject scenarios due to the difficulty of constructing multi-subject training data pairs. Another challenging problem that how to use the signals such as depth, mask, camera, and text prompts to control and edit the subject in the customized video is still less explored. In this paper, we first propose a data construction pipeline, VideoCus-Factory, to produce training data pairs for multi-subject customization from raw videos without labels and control signals such as depth-to-video and mask-to-video pairs. Based on our constructed data, we develop an Image-Video Transfer Mixed (IVTM) training with image editing data to enable instructive editing for the subject in the customized video. Then we propose a diffusion Transformer framework, OmniVCus, with two embedding mechanisms, Lottery Embedding (LE) and Temporally Aligned Embedding (TAE). LE enables inference with more subjects by using the training subjects to activate more frame embeddings. TAE encourages the generation process to extract guidance from temporally aligned control signals by assigning the same frame embeddings to the control and noise tokens. Experiments demonstrate that our method significantly surpasses state-of-the-art methods in both quantitative and qualitative evaluations. Video demos are at our project page: https://caiyuanhao1998.github.io/project/OmniVCus/. Our code, models, data are released at https://github.com/caiyuanhao1998/Open-OmniVCus