Multi-microphone Complex Spectral Mapping for Utterance-wise and Continuous Speech Separation

TL;DR

This paper introduces a multi-microphone complex spectral mapping technique using deep learning for effective speaker separation and dereverberation in reverberant environments, applicable to both offline and online scenarios.

Contribution

It presents a novel deep learning approach that predicts real and imaginary speech components across multiple microphones, integrated with beamforming and post-filtering for improved separation.

Findings

Achieves state-of-the-art results on simulated and real datasets.

Generalizes well from simulated training to real array conditions.

Effective for both offline and online continuous speech separation.

Abstract

We propose multi-microphone complex spectral mapping, a simple way of applying deep learning for time-varying non-linear beamforming, for speaker separation in reverberant conditions. We aim at both speaker separation and dereverberation. Our study first investigates offline utterance-wise speaker separation and then extends to block-online continuous speech separation (CSS). Assuming a fixed array geometry between training and testing, we train deep neural networks (DNN) to predict the real and imaginary (RI) components of target speech at a reference microphone from the RI components of multiple microphones. We then integrate multi-microphone complex spectral mapping with minimum variance distortionless response (MVDR) beamforming and post-filtering to further improve separation, and combine it with frame-level speaker counting for block-online CSS. Although our system is trained on simulated room impulse responses (RIR) based on a fixed number of microphones arranged in a given geometry, it generalizes well to a real array with the same geometry. State-of-the-art separation performance is obtained on the simulated two-talker SMS-WSJ corpus and the real-recorded LibriCSS dataset.