TF-Mamba: A Time-Frequency Network for Sound Source Localization

TL;DR

TF-Mamba is a novel time-frequency neural network architecture designed for sound source localization, effectively fusing spatial features from speech signals to improve accuracy in challenging acoustic environments.

Contribution

Introduces TF-Mamba, a new time-frequency network that enhances sound source localization by integrating bidirectional Mamba modules for spatial feature extraction.

Findings

Significantly outperforms existing SSL methods on simulated datasets.

Effective in real-world acoustic environments.

Demonstrates robustness in challenging conditions.

Abstract

Sound source localization (SSL) determines the position of sound sources using multi-channel audio data. It is commonly used to improve speech enhancement and separation. Extracting spatial features is crucial for SSL, especially in challenging acoustic environments. Recently, a novel structure referred to as Mamba demonstrated notable performance across various sequence-based modalities. This study introduces the Mamba for SSL tasks. We consider the Mamba-based model to analyze spatial features from speech signals by fusing both time and frequency features, and we develop an SSL system called TF-Mamba. This system integrates time and frequency fusion, with Bidirectional Mamba managing both time-wise and frequency-wise processing. We conduct the experiments on the simulated and real datasets. Experiments show that TF-Mamba significantly outperforms other advanced methods. The code will be publicly released in due course.