FoleyGen: Visually-Guided Audio Generation

TL;DR

FoleyGen is a novel system for video-to-audio generation that uses visual features and attention mechanisms to produce synchronized audio, outperforming previous methods on the VGGSound dataset.

Contribution

We introduce FoleyGen, a visual-guided audio generation system utilizing a Transformer conditioned on visual features and novel attention mechanisms for improved synchronization.

Findings

Outperforms previous systems on VGGSound dataset

Uses novel visual attention mechanisms to improve audio-visual alignment

Achieves better objective and human evaluation metrics

Abstract

Recent advancements in audio generation have been spurred by the evolution of large-scale deep learning models and expansive datasets. However, the task of video-to-audio (V2A) generation continues to be a challenge, principally because of the intricate relationship between the high-dimensional visual and auditory data, and the challenges associated with temporal synchronization. In this study, we introduce FoleyGen, an open-domain V2A generation system built on a language modeling paradigm. FoleyGen leverages an off-the-shelf neural audio codec for bidirectional conversion between waveforms and discrete tokens. The generation of audio tokens is facilitated by a single Transformer model, which is conditioned on visual features extracted from a visual encoder. A prevalent problem in V2A generation is the misalignment of generated audio with the visible actions in the video. To address this, we explore three novel visual attention mechanisms. We further undertake an exhaustive evaluation of multiple visual encoders, each pretrained on either single-modal or multi-modal tasks. The experimental results on VGGSound dataset show that our proposed FoleyGen outperforms previous systems across all objective metrics and human evaluations.