Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generation
Yuheng Chen, Qingdong He, Teng Hu, Yuji Wang, Yabiao Wang, Lizhuang Ma, Jiangning Zhang

TL;DR
Omni-Customizer is an end-to-end framework that advances joint audio-video generation by enabling precise multimodal identity customization, fusion, and synchronization, leveraging novel modules and training strategies.
Contribution
The paper introduces Omni-Customizer, a comprehensive multimodal framework with new modules and training methods for improved identity preservation and synchronization in audio-video generation.
Findings
Achieves state-of-the-art performance in visual identity similarity.
Maintains timbre consistency across generated audio.
Ensures precise audio-video synchronization.
Abstract
The landscape of joint audio and video generation has been fundamentally transformed by the advent of powerful foundation models. Despite these strides, achieving cohesive multimodal customization for the simultaneous preservation of visual identities and vocal timbres across multiple interacting subjects remains largely underexplored. To bridge this gap, we present Omni-Customizer, an end-to-end framework targeted at the precise binding and seamless fusion of multimodal identity information. Specifically, we introduce an Omni-Context Fusion (OCF) module that effectively enriches the base textual prompt with dense, multimodal identity cues, along with a Masked TTS Cross-Attention (MTP-CA) mechanism explicitly designed to prevent the severe "speech leakage" problem. Within this architecture, we propose Semantic-Anchored Multimodal RoPE (SA-MRoPE) to anchor visual and audio reference…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
