Towards Spatial Audio Understanding via Question Answering

TL;DR

This paper presents a new question answering framework for spatial audio understanding using FOA signals, combining dataset creation, linguistic diversity, and a baseline model to interpret sound scenes with minimal supervision.

Contribution

It introduces a novel QA-based approach for spatial audio understanding, including dataset curation, linguistic enhancement, and a baseline model trained with scene-level supervision.

Findings

Model achieves performance comparable to fully supervised methods

Enhanced linguistic diversity improves question answering robustness

Dataset and baseline model facilitate future research in spatial audio QA

Abstract

In this paper, we introduce a novel framework for spatial audio understanding of first-order ambisonic (FOA) signals through a question answering (QA) paradigm, aiming to extend the scope of sound event localization and detection (SELD) towards spatial scene understanding and reasoning. First, we curate and release fine-grained spatio-temporal textual descriptions for the STARSS23 dataset using a rule-based approach, and further enhance linguistic diversity using large language model (LLM)-based rephrasing. We also introduce a QA dataset aligned with the STARSS23 scenes, covering various aspects such as event presence, localization, spatial, and temporal relationships. To increase language variety, we again leverage LLMs to generate multiple rephrasings per question. Finally, we develop a baseline spatial audio QA model that takes FOA signals and natural language questions as input and provides answers regarding various occurrences, temporal, and spatial relationships of sound events in the scene formulated as a classification task. Despite being trained solely with scene-level question answering supervision, our model achieves performance that is comparable to a fully supervised sound event localization and detection model trained with frame-level spatiotemporal annotations. The results highlight the potential of language-guided approaches for spatial audio understanding and open new directions for integrating linguistic supervision into spatial scene analysis.