Recursive speech separation for unknown number of speakers

TL;DR

This paper introduces a recursive speech separation method that handles an unknown number of speakers with a single model, using permutation invariant training and speaker detection, achieving state-of-the-art results.

Contribution

It presents a novel recursive separation approach with OR-PIT and speaker number detection, enabling a single model to separate varying speaker counts, including unseen four-speaker mixtures.

Findings

Achieves state-of-the-art results on two- and three-speaker mixtures.

Successfully separates four-speaker mixtures unseen during training.

Accurately detects the number of speakers during recursive separation.

Abstract

In this paper we propose a method of single-channel speaker-independent multi-speaker speech separation for an unknown number of speakers. As opposed to previous works, in which the number of speakers is assumed to be known in advance and speech separation models are specific for the number of speakers, our proposed method can be applied to cases with different numbers of speakers using a single model by recursively separating a speaker. To make the separation model recursively applicable, we propose one-and-rest permutation invariant training (OR-PIT). Evaluation on WSJ0-2mix and WSJ0-3mix datasets show that our proposed method achieves state-of-the-art results for two- and three-speaker mixtures with a single model. Moreover, the same model can separate four-speaker mixture, which was never seen during the training. We further propose the detection of the number of speakers in a mixture during recursive separation and show that this approach can more accurately estimate the number of speakers than detection in advance by using a deep neural network based classifier.