Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation

TL;DR

Kling-Foley is a multimodal diffusion transformer model that synthesizes high-quality, synchronized video-to-audio content, leveraging novel modules for improved alignment, a universal audio codec, and a new benchmark for evaluation.

Contribution

The paper introduces Kling-Foley, a comprehensive multimodal model with innovative modules and a universal audio codec, setting new state-of-the-art results in video-to-audio generation.

Findings

Achieves state-of-the-art performance in audio-visual synchronization.

Effectively models diverse audio types including speech, music, and sound effects.

Provides a new benchmark for evaluating video-to-audio models.

Abstract

We propose Kling-Foley, a large-scale multimodal Video-to-Audio generation model that synthesizes high-quality audio synchronized with video content. In Kling-Foley, we introduce multimodal diffusion transformers to model the interactions between video, audio, and text modalities, and combine it with a visual semantic representation module and an audio-visual synchronization module to enhance alignment capabilities. Specifically, these modules align video conditions with latent audio elements at the frame level, thereby improving semantic alignment and audio-visual synchronization. Together with text conditions, this integrated approach enables precise generation of video-matching sound effects. In addition, we propose a universal latent audio codec that can achieve high-quality modeling in various scenarios such as sound effects, speech, singing, and music. We employ a stereo rendering method that imbues synthesized audio with a spatial presence. At the same time, in order to make up for the incomplete types and annotations of the open-source benchmark, we also open-source an industrial-level benchmark Kling-Audio-Eval. Our experiments show that Kling-Foley trained with the flow matching objective achieves new audio-visual SOTA performance among public models in terms of distribution matching, semantic alignment, temporal alignment and audio quality.