DreamFoley: Scalable VLMs for High-Fidelity Video-to-Audio Generation

TL;DR

DreamFoley introduces a scalable autoregressive model leveraging large vision-language models to generate high-fidelity, synchronized video-to-audio content, addressing previous limitations in audio-visual alignment and quality.

Contribution

The paper presents a novel autoregressive architecture with dual-visual encoders, a residual vector quantization tokenizer, and classifier-free guidance, advancing scalable high-quality video-to-audio generation.

Findings

Achieves promising performance on benchmark datasets.

Effectively balances training efficiency and audio quality.

Provides a new dataset with missing audio-visual textual descriptions.

Abstract

Recent advances in video generation have achieved remarkable improvements in visual content fidelity. However, the absence of synchronized audio severely undermines immersive experience and restricts practical applications of these technologies. To address this challenge, several pioneering works have explored diffusion transformer architectures for generating plausible video-synchronized audio, including Kling-foley, HunyuanVideo-foley and Thinksound. Distinct from existing works, we introduce an autoregressive audio generation architecture (DreamFoley) that harnesses the capabilities of large vision-language models (VLMs) to jointly model sequential interactions among video, audio, and text modalities. Our approach features a dual-visual encoder module that effectively captures both audio-aligned and text-aligned visual features. Additionally, we employ a Residual Vector Quantization audio tokenizer with a delay-pattern generation scheme to balance the trade-off between training efficiency and audio quality. Moreover, we introduce the classifier-free guidance strategy into VLMs to bootstrap generated audio quality. Furthermore, we establish an efficient data production pipeline to scale audio-video-text triple collection. Finally, extensive experiments are conducted to validate the effectiveness of our model, achieving promising performance across popular benchmarks. We hope that the findings in this study provide a strong foundation for future video-to-audio generation research. We also release the previously missing audio-visual textual descriptions from the public benchmark, aiming to facilitate subsequent researchers in conducting more convenient and effective evaluations and comparisons.