Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generation

TL;DR

Omni-Customizer is an end-to-end framework that advances joint audio-video generation by enabling precise multimodal identity customization, fusion, and synchronization, leveraging novel modules and training strategies.

Contribution

The paper introduces Omni-Customizer, a comprehensive multimodal framework with new modules and training methods for improved identity preservation and synchronization in audio-video generation.

Findings

Achieves state-of-the-art performance in visual identity similarity.

Maintains timbre consistency across generated audio.

Ensures precise audio-video synchronization.

Abstract

The landscape of joint audio and video generation has been fundamentally transformed by the advent of powerful foundation models. Despite these strides, achieving cohesive multimodal customization for the simultaneous preservation of visual identities and vocal timbres across multiple interacting subjects remains largely underexplored. To bridge this gap, we present Omni-Customizer, an end-to-end framework targeted at the precise binding and seamless fusion of multimodal identity information. Specifically, we introduce an Omni-Context Fusion (OCF) module that effectively enriches the base textual prompt with dense, multimodal identity cues, along with a Masked TTS Cross-Attention (MTP-CA) mechanism explicitly designed to prevent the severe "speech leakage" problem. Within this architecture, we propose Semantic-Anchored Multimodal RoPE (SA-MRoPE) to anchor visual and audio reference tokens, along with TTS embeddings, to their corresponding semantic descriptions, enabling structured multimodal fusion and robust identity binding. Furthermore, we devise a comprehensive training strategy that incorporates interleaved audio-video scheduling to rapidly adapt the audio branch to multilingual scenarios without degrading foundational priors, and a progressive in-pair to cross-pair curriculum to facilitate the learning of high-level and robust identity features. Extensive experiments demonstrate that Omni-Customizer achieves state-of-the-art performance in dual-modal customized generation, excelling across visual identity similarity, timbre consistency, precise audio-video synchronization, and overall video-audio fidelity.