LuSeeL: Language-queried Binaural Universal Sound Event Extraction and Localization

TL;DR

This paper introduces LuSeeL, a binaural audio model that extracts sound events based on language descriptions and estimates their spatial location, leveraging 3D spatial cues for improved accuracy in complex auditory scenes.

Contribution

We propose a novel language-driven binaural sound extraction network that jointly predicts sound source location, enhancing extraction performance with spatial cues.

Findings

LuSeeL outperforms single-channel and uni-task baselines.

The model effectively leverages binaural spatial cues.

Joint extraction and localization improve accuracy.

Abstract

Most universal sound extraction algorithms focus on isolating a target sound event from single-channel audio mixtures. However, the real world is three-dimensional, and binaural audio, which mimics human hearing, can capture richer spatial information, including sound source location. This spatial context is crucial for understanding and modeling complex auditory scenes, as it inherently informs sound detection and extraction. In this work, we propose a language-driven universal sound extraction network that isolates text-described sound events from binaural mixtures by effectively leveraging the spatial cues present in binaural signals. Additionally, we jointly predict the direction of arrival (DoA) of the target sound using spatial features from the extraction network. This dual-task approach exploits complementary location information to improve extraction performance while enabling accurate DoA estimation. Experimental results on the in-the-wild AudioCaps dataset show that our proposed LuSeeL model significantly outperforms single-channel and uni-task baselines.