AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation

TL;DR

AV-DiT introduces an efficient audio-visual diffusion transformer that leverages a shared backbone with lightweight adapters, enabling high-quality joint audio-visual content generation with reduced complexity and parameters.

Contribution

The paper presents a novel shared backbone diffusion transformer with modality-specific adapters for efficient joint audio-visual generation, achieving state-of-the-art results with fewer parameters.

Findings

State-of-the-art performance on AIST++ and Landscape datasets.

Significantly fewer tunable parameters compared to existing methods.

A shared image generative backbone suffices for joint audio-visual generation.

Abstract

Recent Diffusion Transformers (DiTs) have shown impressive capabilities in generating high-quality single-modality content, including images, videos, and audio. However, it is still under-explored whether the transformer-based diffuser can efficiently denoise the Gaussian noises towards superb multimodal content creation. To bridge this gap, we introduce AV-DiT, a novel and efficient audio-visual diffusion transformer designed to generate high-quality, realistic videos with both visual and audio tracks. To minimize model complexity and computational costs, AV-DiT utilizes a shared DiT backbone pre-trained on image-only data, with only lightweight, newly inserted adapters being trainable. This shared backbone facilitates both audio and video generation. Specifically, the video branch incorporates a trainable temporal attention layer into a frozen pre-trained DiT block for temporal consistency. Additionally, a small number of trainable parameters adapt the image-based DiT block for audio generation. An extra shared DiT block, equipped with lightweight parameters, facilitates feature interaction between audio and visual modalities, ensuring alignment. Extensive experiments on the AIST++ and Landscape datasets demonstrate that AV-DiT achieves state-of-the-art performance in joint audio-visual generation with significantly fewer tunable parameters. Furthermore, our results highlight that a single shared image generative backbone with modality-specific adaptations is sufficient for constructing a joint audio-video generator. Our source code and pre-trained models will be released.