AuralNet: Hierarchical Attention-based 3D Binaural Localization of Overlapping Speakers

TL;DR

AuralNet is a hierarchical attention-based neural network that accurately localizes multiple overlapping sound sources in 3D space using binaural signals, even in noisy and reverberant environments.

Contribution

It introduces a novel hierarchical architecture with attention mechanisms for multi-source 3D localization without prior source count knowledge.

Findings

Outperforms recent localization methods in noisy-reverberant settings

Effectively localizes overlapping sources in azimuth and elevation

Robust to environmental noise and reverberation

Abstract

We propose AuralNet, a novel 3D multi-source binaural sound source localization approach that localizes overlapping sources in both azimuth and elevation without prior knowledge of the number of sources. AuralNet employs a gated coarse-tofine architecture, combining a coarse classification stage with a fine-grained regression stage, allowing for flexible spatial resolution through sector partitioning. The model incorporates a multi-head self-attention mechanism to capture spatial cues in binaural signals, enhancing robustness in noisy-reverberant environments. A masked multi-task loss function is designed to jointly optimize sound detection, azimuth, and elevation estimation. Extensive experiments in noisy-reverberant conditions demonstrate the superiority of AuralNet over recent methods