SOAF: Scene Occlusion-aware Neural Acoustic Field

TL;DR

This paper introduces SOAF, a novel scene occlusion-aware neural acoustic model that improves sound synthesis accuracy in indoor environments by incorporating room geometry and wall occlusion effects.

Contribution

The paper presents a new approach called SOAF that models room geometry and occlusions for more accurate sound propagation in novel view audio-visual synthesis.

Findings

Outperforms previous methods on RWAVS and SoundSpaces datasets.

Effectively models wall occlusions and room geometry impacts.

Generates high-quality binaural audio for arbitrary viewpoints.

Abstract

This paper tackles the problem of novel view audio-visual synthesis along an arbitrary trajectory in an indoor scene, given the audio-video recordings from other known trajectories of the scene. Existing methods often overlook the effect of room geometry, particularly wall occlusions on sound propagation, making them less accurate in multi-room environments. In this work, we propose a new approach called Scene Occlusion-aware Acoustic Field (SOAF) for accurate sound generation. Our approach derives a global prior for the sound field using distance-aware parametric sound-propagation modeling and then transforms it based on the scene structure learned from the input video. We extract features from the local acoustic field centered at the receiver using a Fibonacci Sphere to generate binaural audio for novel views with a direction-aware attention mechanism. Extensive experiments on the real dataset RWAVS and the synthetic dataset SoundSpaces demonstrate that our method outperforms previous state-of-the-art techniques in audio generation.