Scene-Aware Audio for 360\textdegree{} Videos

TL;DR

This paper introduces a novel method for creating realistic, scene-aware 360-degree spatial audio for indoor videos using only a mono microphone and speaker, by synthesizing impulse responses based on room acoustics.

Contribution

The method synthesizes directional impulse responses by combining simulated early reverberation with measured late reverberation, enabling realistic spatial audio in 360-degree videos with minimal equipment.

Findings

Synthesized audio closely matches ambisonic recordings.

Effective in indoor scenes with diffuse reverberation.

Applicable to typical consumer-grade recording setups.

Abstract

Although 360\textdegree{} cameras ease the capture of panoramic footage, it remains challenging to add realistic 360\textdegree{} audio that blends into the captured scene and is synchronized with the camera motion. We present a method for adding scene-aware spatial audio to 360\textdegree{} videos in typical indoor scenes, using only a conventional mono-channel microphone and a speaker. We observe that the late reverberation of a room's impulse response is usually diffuse spatially and directionally. Exploiting this fact, we propose a method that synthesizes the directional impulse response between any source and listening locations by combining a synthesized early reverberation part and a measured late reverberation tail. The early reverberation is simulated using a geometric acoustic simulation and then enhanced using a frequency modulation method to capture room resonances. The late reverberation is extracted from a recorded impulse response, with a carefully chosen time duration that separates out the late reverberation from the early reverberation. In our validations, we show that our synthesized spatial audio matches closely with recordings using ambisonic microphones. Lastly, we demonstrate the strength of our method in several applications.