Sonic4D: Spatial Audio Generation for Immersive 4D Scene Exploration

TL;DR

Sonic4D introduces a framework that synthesizes spatial audio aligned with 4D scene visualizations, significantly enhancing immersive audiovisual experiences by localizing sound sources and generating realistic spatial audio in a training-free manner.

Contribution

The paper presents a novel, training-free method for generating spatial audio synchronized with 4D scenes, integrating visual grounding and physics-based audio synthesis.

Findings

Produces realistic spatial audio consistent with 4D scenes

Enhances immersion in audiovisual exploration

Operates without additional training data

Abstract

Recent advancements in 4D generation have demonstrated its remarkable capability in synthesizing photorealistic renderings of dynamic 3D scenes. However, despite achieving impressive visual performance, almost all existing methods overlook the generation of spatial audio aligned with the corresponding 4D scenes, posing a significant limitation to truly immersive audiovisual experiences. To mitigate this issue, we propose Sonic4D, a novel framework that enables spatial audio generation for immersive exploration of 4D scenes. Specifically, our method is composed of three stages: 1) To capture both the dynamic visual content and raw auditory information from a monocular video, we first employ pre-trained expert models to generate the 4D scene and its corresponding monaural audio. 2) Subsequently, to transform the monaural audio into spatial audio, we localize and track the sound sources within the 4D scene, where their 3D spatial coordinates at different timestamps are estimated via a pixel-level visual grounding strategy. 3) Based on the estimated sound source locations, we further synthesize plausible spatial audio that varies across different viewpoints and timestamps using physics-based simulation. Extensive experiments have demonstrated that our proposed method generates realistic spatial audio consistent with the synthesized 4D scene in a training-free manner, significantly enhancing the immersive experience for users. Generated audio and video examples are available at https://x-drunker.github.io/Sonic4D-project-page.