Acoustic Field Video for Multimodal Scene Understanding

TL;DR

This paper introduces acoustic field video as a new multimodal input for scene understanding, demonstrating significant improvements in visual-language model performance by incorporating spatial sound data from low-cost microphone arrays.

Contribution

It presents a novel acoustic field video representation and evaluates its effectiveness in enhancing scene understanding in vision-language models.

Findings

Increased accuracy from 38.3% to 67.4% with acoustic data

Acoustic field video provides a new perceptual dimension

Spatial sound improves multimodal reasoning

Abstract

We introduce and explore a new multimodal input representation for vision-language models: acoustic field video. Unlike conventional video (RGB with stereo/mono audio), our video stream provides a spatially grounded visualization of sound intensity across a scene, offering a new and powerful dimension of perceptual understanding. Our real-time pipeline uses low-cost beamforming microphone arrays that are already common in smart speakers and increasingly present in robotics and XR headsets, yet this sensing capability remains unutilized for scene understanding. To assess the value of spatial acoustic information, we constructed an evaluation set of 402 question-answer scenes, comparing a state-of-the-art VLM given conventional video with and without paired acoustic field video. Results show a clear and consistent improvement when incorporating spatial acoustic data; the VLM we test improves from 38.3% correct to 67.4%. Our findings highlight that many everyday scene understanding tasks remain underconstrained when relying solely on visual and audio input, and that acoustic field data provides a promising and practical direction for multimodal reasoning. A video demo is available at https://daehwakim.com/seeingsound