AudioSpa: Spatializing Sound Events with Text

TL;DR

AudioSpa is an innovative system that generates binaural spatial audio from text and monaural references, enhancing immersive sound experiences by accurately localizing sound sources using multimodal learning and data augmentation.

Contribution

This work introduces AudioSpa, the first end-to-end model for text-guided binaural audio generation with spatial localization capabilities, utilizing large language models and novel fusion techniques.

Findings

Achieves accurate sound source localization from text descriptions.

Demonstrates competitive performance in localization accuracy.

Employs data augmentation to improve spatialization diversity.

Abstract

Text-to-audio (TTA) systems have recently demonstrated strong performance in synthesizing monaural audio from text. However, the task of generating binaural spatial audio from text, which provides a more immersive auditory experience by incorporating the sense of spatiality, have not been explored yet. In this work, we introduce text-guided binaural audio generation. As an early effort, we focus on the scenario where a monaural reference audio is given additionally. The core problem is to associate specific sound events with their directions, thereby creating binaural spatial audio. The challenge lies in the complexity of textual descriptions and the limited availability of single-source sound event datasets. To address this, we propose AudioSpa, an end-to-end model that applies large language models to process both acoustic and textual information. We employ fusion multi-head attention (FMHA) to integrate text tokens, which enhances the generation capability of the multimodal learning. Additionally, we propose a binaural source localization model to assess the quality of the generated audio. Finally, we design a data augmentation strategy to generate diverse datasets, which enables the model to spatialize sound events across various spatial positions. Experimental results demonstrate that our model is able to put sounds at the specified locations accurately. It achieves competitive performance in both localization accuracy and signal distortion. Our demonstrations are available at https://linfeng-feng.github.io/AudioSpa-demo.