ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Model

TL;DR

ImmerseDiffusion is a novel generative model that creates 3D immersive soundscapes conditioned on spatial, temporal, and environmental parameters, using a latent diffusion approach for high-quality spatial audio synthesis.

Contribution

The paper introduces a new end-to-end generative system for spatial audio that combines a spatial audio codec, a latent diffusion model, and a CLAP-style encoder, enabling flexible and conditioned 3D soundscape generation.

Findings

Model achieves high spatial fidelity in generated audio.

Two modes, descriptive and parametric, effectively produce spatial soundscapes.

Metrics show the model's outputs are consistent with user conditions.

Abstract

We introduce ImmerseDiffusion, an end-to-end generative audio model that produces 3D immersive soundscapes conditioned on the spatial, temporal, and environmental conditions of sound objects. ImmerseDiffusion is trained to generate first-order ambisonics (FOA) audio, which is a conventional spatial audio format comprising four channels that can be rendered to multichannel spatial output. The proposed generative system is composed of a spatial audio codec that maps FOA audio to latent components, a latent diffusion model trained based on various user input types, namely, text prompts, spatial, temporal and environmental acoustic parameters, and optionally a spatial audio and text encoder trained in a Contrastive Language and Audio Pretraining (CLAP) style. We propose metrics to evaluate the quality and spatial adherence of the generated spatial audio. Finally, we assess the model performance in terms of generation quality and spatial conformance, comparing the two proposed modes: ``descriptive", which uses spatial text prompts) and ``parametric", which uses non-spatial text prompts and spatial parameters. Our evaluations demonstrate promising results that are consistent with the user conditions and reflect reliable spatial fidelity.