Gotta Hear Them All: Towards Sound Source Aware Audio Generation

TL;DR

This paper introduces SS2A, a novel sound source-aware audio generator that improves audio synthesis by locally perceiving sound sources and disambiguating their semantics, leading to more immersive and controllable audio generation.

Contribution

The paper proposes a new sound source-aware framework with a curated dataset and novel metrics, advancing the state-of-the-art in image-to-audio and video-to-audio synthesis.

Findings

Achieves state-of-the-art results in image-to-audio tasks.

Demonstrates intuitive control over audio synthesis via visual and textual inputs.

Performs competitively in video-to-audio tasks with simple temporal aggregation.

Abstract

Audio synthesis has broad applications in multimedia. Recent advancements have made it possible to generate relevant audios from inputs describing an audio scene, such as images or texts. However, the immersiveness and expressiveness of the generation are limited. One possible problem is that existing methods solely rely on the global scene and overlook details of local sounding objects (i.e., sound sources). To address this issue, we propose a Sound Source-Aware Audio (SS2A) generator. SS2A is able to locally perceive multimodal sound sources from a scene with visual detection and cross-modality translation. It then contrastively learns a Cross-Modal Sound Source (CMSS) Manifold to semantically disambiguate each source. Finally, we attentively mix their CMSS semantics into a rich audio representation, from which a pretrained audio generator outputs the sound. To model the CMSS manifold, we curate a novel single-sound-source visual-audio dataset VGGS3 from VGGSound. We also design a Sound Source Matching Score to clearly measure localized audio relevance. With the effectiveness of explicit sound source modeling, SS2A achieves state-of-the-art performance in extensive image-to-audio tasks. We also qualitatively demonstrate SS2A's ability to achieve intuitive synthesis control by compositing vision, text, and audio conditions. Furthermore, we show that our sound source modeling can achieve competitive video-to-audio performance with a straightforward temporal aggregation mechanism.