An Initial Exploration: Learning to Generate Realistic Audio for Silent Video

TL;DR

This paper explores deep learning models, especially transformer architectures, to generate realistic audio from silent video, aiming to replicate Foley art through automated means.

Contribution

It introduces a deep-learning framework for audio generation from silent video, comparing multiple architectures and highlighting the effectiveness of transformers.

Findings

Transformers best match low-frequency audio to visual cues.

Models struggle to generate nuanced waveforms.

Deep-fusion CNNs and dilated Wavenet are less effective.

Abstract

Generating realistic audio effects for movies and other media is a challenging task that is accomplished today primarily through physical techniques known as Foley art. Foley artists create sounds with common objects (e.g., boxing gloves, broken glass) in time with video as it is playing to generate captivating audio tracks. In this work, we aim to develop a deep-learning based framework that does much the same - observes video in it's natural sequence and generates realistic audio to accompany it. Notably, we have reason to believe this is achievable due to advancements in realistic audio generation techniques conditioned on other inputs (e.g., Wavenet conditioned on text). We explore several different model architectures to accomplish this task that process both previously-generated audio and video context. These include deep-fusion CNN, dilated Wavenet CNN with visual context, and transformer-based architectures. We find that the transformer-based architecture yields the most promising results, matching low-frequencies to visual patterns effectively, but failing to generate more nuanced waveforms.