Read, Watch and Scream! Sound Generation from Text and Video

TL;DR

This paper introduces a novel multimodal generative approach called urs that combines video and text cues to generate controllable, high-quality audio, improving flexibility and efficiency over existing methods.

Contribution

The method uniquely integrates video-based structural cues with text prompts to enhance audio generation control and efficiency in multimodal diffusion models.

Findings

Outperforms existing models in audio quality and controllability.

Enables user adjustments of energy, environment, and sound sources.

Demonstrates improved training efficiency with large triplet data.

Abstract

Despite the impressive progress of multimodal generative models, video-to-audio generation still suffers from limited performance and limits the flexibility to prioritize sound synthesis for specific objects within the scene. Conversely, text-to-audio generation methods generate high-quality audio but pose challenges in ensuring comprehensive scene depiction and time-varying control. To tackle these challenges, we propose a novel video-and-text-to-audio generation method, called \ours, where video serves as a conditional control for a text-to-audio generation model. Especially, our method estimates the structural information of sound (namely, energy) from the video while receiving key content cues from a user prompt. We employ a well-performing text-to-audio model to consolidate the video control, which is much more efficient for training multimodal diffusion models with massive triplet-paired (audio-video-text) data. In addition, by separating the generative components of audio, it becomes a more flexible system that allows users to freely adjust the energy, surrounding environment, and primary sound source according to their preferences. Experimental results demonstrate that our method shows superiority in terms of quality, controllability, and training efficiency. Code and demo are available at https://naver-ai.github.io/rewas.