DasFormer: Deep Alternating Spectrogram Transformer for Multi/Single-Channel Speech Separation

TL;DR

DasFormer is a unified deep learning model that effectively handles both multi-channel and single-channel speech separation in reverberant environments by using alternating attention mechanisms on spectrograms.

Contribution

It introduces a simple, unified architecture that processes spectral and spatial information with alternating self-attention modules for speech separation.

Findings

Outperforms current state-of-the-art models in multi-channel speech separation

Achieves single-channel state-of-the-art results in reverberant scenarios

Demonstrates strong modeling of time-frequency representations

Abstract

For the task of speech separation, previous study usually treats multi-channel and single-channel scenarios as two research tracks with specialized solutions developed respectively. Instead, we propose a simple and unified architecture - DasFormer (Deep alternating spectrogram transFormer) to handle both of them in the challenging reverberant environments. Unlike frame-wise sequence modeling, each TF-bin in the spectrogram is assigned with an embedding encoding spectral and spatial information. With such input, DasFormer is then formed by multiple repetition of simple blocks each of which integrates 1) two multi-head self-attention (MHSA) modules alternately processing within each frequency bin & temporal frame of the spectrogram 2) MBConv before each MHSA for modeling local features on the spectrogram. Experiments show that DasFormer has a powerful ability to model the time-frequency representation, whose performance far exceeds the current SOTA models in multi-channel speech separation, and also achieves single-channel SOTA in the more challenging yet realistic reverberation scenario.