Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation

TL;DR

This paper introduces a novel multimodal diffusion-based approach for controllable stereo spatial audio generation from text, images, and retrieval, addressing challenges of complexity, data costs, and stability.

Contribution

It presents the first large-scale, simulation-based dataset and a spatial-aware diffusion model that enables accurate, immersive, and controllable spatial audio synthesis from multiple modalities.

Findings

Effective spatial audio generation from text and images

Outperforms existing methods in realism and controllability

Demonstrates physical plausibility in generated spatial sound

Abstract

Recently, diffusion models have achieved great success in mono-channel audio generation. However, when it comes to stereo audio generation, the soundscapes often have a complex scene of multiple objects and directions. Controlling stereo audio with spatial contexts remains challenging due to high data costs and unstable generative models. To the best of our knowledge, this work represents the first attempt to address these issues. We first construct a large-scale, simulation-based, and GPT-assisted dataset, BEWO-1M, with abundant soundscapes and descriptions even including moving and multiple sources. Beyond text modality, we have also acquired a set of images and rationally paired stereo audios through retrieval to advance multimodal generation. Existing audio generation models tend to generate rather random and indistinct spatial audio. To provide accurate guidance for Latent Diffusion Models, we introduce the SpatialSonic model utilizing spatial-aware encoders and azimuth state matrices to reveal reasonable spatial guidance. By leveraging spatial guidance, our model not only achieves the objective of generating immersive and controllable spatial audio from text but also extends to other modalities as the pioneer attempt. Finally, under fair settings, we conduct subjective and objective evaluations on simulated and real-world data to compare our approach with prevailing methods. The results demonstrate the effectiveness of our method, highlighting its capability to generate spatial audio that adheres to physical rules.