Physics-Aware Novel-View Acoustic Synthesis with Vision-Language Priors and 3D Acoustic Environment Modeling

TL;DR

Phys-NVAS introduces a physics-aware framework for novel-view acoustic synthesis that combines 3D environment modeling with vision-language semantic priors, significantly improving spatial audio realism and physical consistency.

Contribution

This work is the first to integrate physical environment modeling with vision-language priors for NVAS, enhancing spatial and semantic sound synthesis capabilities.

Findings

Improved realism in binaural audio generation.

Enhanced physical consistency of synthesized spatial sounds.

Effective integration of geometry and semantic cues.

Abstract

Spatial audio is essential for immersive experiences, yet novel-view acoustic synthesis (NVAS) remains challenging due to complex physical phenomena such as reflection, diffraction, and material absorption. Existing methods based on single-view or panoramic inputs improve spatial fidelity but fail to capture global geometry and semantic cues such as object layout and material properties. To address this, we propose Phys-NVAS, the first physics-aware NVAS framework that integrates spatial geometry modeling with vision-language semantic priors. A global 3D acoustic environment is reconstructed from multi-view images and depth maps to estimate room size and shape, enhancing spatial awareness of sound propagation. Meanwhile, a vision-language model extracts physics-aware priors of objects, layouts, and materials, capturing absorption and reflection beyond geometry. An acoustic feature fusion adapter unifies these cues into a physics-aware representation for binaural generation. Experiments on RWAVS demonstrate that Phys-NVAS yields binaural audio with improved realism and physical consistency.