DreamID-Omni: Unified Framework for Controllable Human-Centric Audio-Video Generation

TL;DR

DreamID-Omni introduces a unified, controllable framework for human-centric audio-video generation, effectively managing multiple identities and voices with novel disentanglement and training strategies, achieving state-of-the-art results.

Contribution

The paper presents a novel symmetric conditional diffusion transformer, dual-level disentanglement, and multi-task progressive training for comprehensive controllable audio-video synthesis.

Findings

Achieves state-of-the-art performance across multiple audio-visual tasks.

Effectively disentangles identities and voice timbres in multi-person scenarios.

Outperforms leading commercial models in quality and consistency.

Abstract

Recent advancements in foundation models have revolutionized joint audio-video generation. However, existing approaches typically treat human-centric tasks including reference-based audio-video generation (R2AV), video editing (RV2AV) and audio-driven video animation (RA2V) as isolated objectives. Furthermore, achieving precise, disentangled control over multiple character identities and voice timbres within a single framework remains an open challenge. In this paper, we propose DreamID-Omni, a unified framework for controllable human-centric audio-video generation. Specifically, we design a Symmetric Conditional Diffusion Transformer that integrates heterogeneous conditioning signals via a symmetric conditional injection scheme. To resolve the pervasive identity-timbre binding failures and speaker confusion in multi-person scenarios, we introduce a Dual-Level Disentanglement strategy: Synchronized RoPE at the signal level to ensure rigid attention-space binding, and Structured Captions at the semantic level to establish explicit attribute-subject mappings. Furthermore, we devise a Multi-Task Progressive Training scheme that leverages weakly-constrained generative priors to regularize strongly-constrained tasks, preventing overfitting and harmonizing disparate objectives. Extensive experiments demonstrate that DreamID-Omni achieves comprehensive state-of-the-art performance across video, audio, and audio-visual consistency, even outperforming leading proprietary commercial models. We will release our code to bridge the gap between academic research and commercial-grade applications.