Rendering Spatial Sound for Interoperable Experiences in the Audio Metaverse

TL;DR

This paper presents a practical design for parametric 6-DOF object-based binaural audio engines to create realistic, immersive spatial sound experiences in the Metaverse, addressing virtual and real environment interactions.

Contribution

It introduces a novel approach for developing interoperable, real-time audio engines that incorporate room acoustics and object properties for immersive Metaverse applications.

Findings

Addresses effects of reverberation, reflectors, and obstacles in virtual and real environments.

Proposes a scene description model for interoperable, multi-platform applications.

Combines pre-computed and real-time acoustic propagation for realistic sound rendering.

Abstract

Interactive audio spatialization technology previously developed for video game authoring and rendering has evolved into an essential component of platforms enabling shared immersive virtual experiences for future co-presence, remote collaboration and entertainment applications. New wearable virtual and augmented reality displays employ real-time binaural audio computing engines rendering multiple digital objects and supporting the free navigation of networked participants or their avatars through a juxtaposition of environments, real and virtual, often referred to as the Metaverse. These applications require a parametric audio scene programming interface to facilitate the creation and deployment of shared, dynamic and realistic virtual 3D worlds on mobile computing platforms and remote servers. We propose a practical approach for designing parametric 6-degree-of-freedom object-based interactive audio engines to deliver the perceptually relevant binaural cues necessary for audio/visual and virtual/real congruence in Metaverse experiences. We address the effects of room reverberation, acoustic reflectors, and obstacles in both the virtual and real environments, and discuss how such effects may be driven by combinations of pre-computed and real-time acoustic propagation solvers. We envision an open scene description model distilled to facilitate the development of interoperable applications distributed across multiple platforms, where each audio object represents, to the user, a natural sound source having controllable distance, size, orientation, and acoustic radiation properties.