Apollo: Unified Multi-Task Audio-Video Joint Generation

TL;DR

Apollo introduces a unified multi-task framework for high-quality, temporally aligned audio-video generation, leveraging novel architecture, training strategies, and a large-scale dataset to outperform prior methods and enhance generalization.

Contribution

The paper presents a novel unified model architecture, a progressive training regime, and a large-scale dataset for improved audio-video joint generation, addressing key challenges in synchronization and generalization.

Findings

Achieves tight audio-visual alignment and scalability.

Outperforms prior methods significantly across tasks.

Generalizes robustly to out-of-distribution data.

Abstract

Audio-video joint generation has progressed rapidly, yet substantial challenges still remain. Non-commercial approaches still suffer audio-visual asynchrony, poor lip-speech alignment, and unimodal degradation, which can be stemmed from weak audio-visual correspondence modeling, limited generalization, and scarce high-quality dense-caption data. To address these issues, we introduce Apollo and delve into three axes--model architecture, training strategy, and data curation. Architecturally, we adopt a single-tower design with unified DiT blocks and an Omni-Full Attention mechanism, achieving tight audio-visual alignment and strong scalability. Training-wise, we adopt a progressive multitask regime--random modality masking to joint optimization across tasks, and a multistage curriculum, yielding robust representations, strengthening A-V aligned world knowledge, and preventing unimodal collapse. For datasets, we present the first large-scale audio-video dataset with dense captions, and introduce a novel automated data-construction pipeline which annotates and filters millions of diverse, high-quality, strictly aligned audio-video-caption triplets. Building on this, Apollo scales to large datasets, delivering high-fidelity, semantically and temporally aligned, instruction-following generation in both joint and unimodal settings while generalizing robustly to out-of-distribution scenarios. Across tasks, it substantially outperforms prior methods by a large margin and achieves performance comparable to Veo 3, offering a unified, scalable path toward next-generation audio-video synthesis.