Listen2Scene: Interactive material-aware binaural sound propagation for reconstructed 3D scenes

TL;DR

Listen2Scene introduces a neural-network-based binaural sound propagation method for VR/AR that efficiently generates realistic acoustic effects for reconstructed 3D indoor environments, enhancing audio realism in real-time.

Contribution

It presents a novel graph neural network and CGAN framework for scene-aware acoustic effect generation, handling mesh artifacts and enabling fast, plausible binaural audio rendering.

Findings

Achieves acoustic effect generation in 0.1 ms on GPU.

Outperforms prior geometric and learning-based methods in plausibility.

Validated with perceptual and quantitative evaluations.

Abstract

We present an end-to-end binaural audio rendering approach (Listen2Scene) for virtual reality (VR) and augmented reality (AR) applications. We propose a novel neural-network-based binaural sound propagation method to generate acoustic effects for indoor 3D models of real environments. Any clean audio or dry audio can be convolved with the generated acoustic effects to render audio corresponding to the real environment. We propose a graph neural network that uses both the material and the topology information of the 3D scenes and generates a scene latent vector. Moreover, we use a conditional generative adversarial network (CGAN) to generate acoustic effects from the scene latent vector. Our network can handle holes or other artifacts in the reconstructed 3D mesh model. We present an efficient cost function for the generator network to incorporate spatial audio effects. Given the source and the listener position, our learning-based binaural sound propagation approach can generate an acoustic effect in 0.1 milliseconds on an NVIDIA GeForce RTX 2080 Ti GPU. We have evaluated the accuracy of our approach with binaural acoustic effects generated using an interactive geometric sound propagation algorithm and captured real acoustic effects / real-world recordings. We also performed a perceptual evaluation and observed that the audio rendered by our approach is more plausible than audio rendered using prior learning-based and geometric-based sound propagation algorithms. We quantitatively evaluated the accuracy of our approach using statistical acoustic parameters, and energy decay curves. The demo videos, code and dataset are available online (https://anton-jeran.github.io/Listen2Scene/).