ALIVE: Animate Your World with Lifelike Audio-Video Generation

TL;DR

ALIVE is a versatile audio-video generation model that adapts pretrained Text-to-Video models for synchronized, animated, and high-quality audio-visual content creation, outperforming existing open-source and commercial solutions.

Contribution

The paper introduces ALIVE, a novel model that extends T2V models to support synchronized audio-video generation and animation, with new architecture components and a comprehensive data pipeline.

Findings

Outperforms open-source models in audio-video generation tasks.

Matches or surpasses state-of-the-art commercial solutions.

Introduces a new benchmark for model evaluation.

Abstract

Video generation is rapidly evolving towards unified audio-video generation. In this paper, we present ALIVE, a generation model that adapts a pretrained Text-to-Video (T2V) model to Sora-style audio-video generation and animation. In particular, the model unlocks the Text-to-Video&Audio (T2VA) and Reference-to-Video&Audio (animation) capabilities compared to the T2V foundation models. To support the audio-visual synchronization and reference animation, we augment the popular MMDiT architecture with a joint audio-video branch which includes TA-CrossAttn for temporally-aligned cross-modal fusion and UniTemp-RoPE for precise audio-visual alignment. Meanwhile, a comprehensive data pipeline consisting of audio-video captioning, quality control, etc., is carefully designed to collect high-quality finetuning data. Additionally, we introduce a new benchmark to perform a comprehensive model test and comparison. After continue pretraining and finetuning on million-level high-quality data, ALIVE demonstrates outstanding performance, consistently outperforming open-source models and matching or surpassing state-of-the-art commercial solutions. With detailed recipes and benchmarks, we hope ALIVE helps the community develop audio-video generation models more efficiently. Official page: https://github.com/FoundationVision/Alive.