Asymmetric Encoder-Decoder Based on Time-Frequency Correlation for Speech Separation

TL;DR

This paper introduces SR-CorrNet, an asymmetric encoder-decoder model with a structured correlation-to-filter approach and dynamic speaker adaptation, significantly improving speech separation in complex acoustic environments.

Contribution

It proposes a novel asymmetric TF encoder-decoder architecture with correlation-based filter estimation and dynamic speaker number adaptation, advancing speech separation methods.

Findings

Consistent improvements on WSJ0-Mix, WHAMR!, and LibriCSS datasets.

Effective in both single- and multi-channel settings.

Enhances separation quality in noisy and reverberant conditions.

Abstract

Speech separation in realistic acoustic environments remains challenging because overlapping speakers, background noise, and reverberation must be resolved simultaneously. Although recent time-frequency (TF) domain models have shown strong performance, most still rely on late-split architectures, where speaker disentanglement is deferred to the final stage, creating an information bottleneck and weakening discriminability under adverse conditions. To address this issue, we propose SR-CorrNet, an asymmetric encoder-decoder framework that introduces the separation-reconstruction (SepRe) strategy into a TF dual-path backbone. The encoder performs coarse separation from mixture observations, while the weight-shared decoder progressively reconstructs speaker-discriminative features with cross-speaker interaction, enabling stage-wise refinement. To complement this architecture, we formulate speech separation as a structured correlation-to-filter problem: spatio-spectro-temporal correlations computed from the observations are used as input features, and the corresponding deep filters are estimated to recover target signals. We further incorporate an attractor-based dynamic split module to adapt the number of output streams to the actual speaker configuration. Experimental results on WSJ0-{2,3,4,5}Mix, WHAMR!, and LibriCSS demonstrate consistent improvements across anechoic, noisy-reverberant, and real-recorded conditions in both single- and multi-channel settings, highlighting the effectiveness of TF-domain SepRe with correlation-based filter estimation for speech separation.