Can Large Language Models Understand Spatial Audio?

TL;DR

This paper demonstrates that large language models can be adapted to understand spatial audio, improving tasks like sound localization and speech recognition in 3D environments, thus enabling more intelligent audio-aware agents.

Contribution

The paper introduces methods for LLMs to interpret spatial audio cues, significantly improving localization accuracy and enabling new audio-based interaction capabilities.

Findings

Achieved 2.70° MAE in sound source localization, surpassing previous benchmarks.

Enhanced far-field speech recognition accuracy using spatial cues.

Enabled LSE with text prompts to focus on specific sound directions.

Abstract

This paper explores enabling large language models (LLMs) to understand spatial information from multichannel audio, a skill currently lacking in auditory LLMs. By leveraging LLMs' advanced cognitive and inferential abilities, the aim is to enhance understanding of 3D environments via audio. We study 3 spatial audio tasks: sound source localization (SSL), far-field speech recognition (FSR), and localisation-informed speech extraction (LSE), achieving notable progress in each task. For SSL, our approach achieves an MAE of $2.70^{\circ}$ on the Spatial LibriSpeech dataset, substantially surpassing the prior benchmark of about $6.60^{\circ}$. Moreover, our model can employ spatial cues to improve FSR accuracy and execute LSE by selectively attending to sounds originating from a specified direction via text prompts, even amidst overlapping speech. These findings highlight the potential of adapting LLMs to grasp physical audio concepts, paving the way for LLM-based agents in 3D environments.