Multi-Decoder DPRNN: High Accuracy Source Counting and Separation

TL;DR

This paper introduces Multi-Decoder DPRNN, an end-to-end model that accurately counts and separates speech sources with an unknown number of speakers, improving counting accuracy and maintaining competitive separation quality.

Contribution

It extends the MulCat model with additional heads for speaker counting and signal reconstruction, and proposes a new evaluation metric for variable speaker scenarios.

Findings

Outperforms state-of-the-art in speaker counting

Remains competitive in separation quality

Effective on mixtures with up to five speakers

Abstract

We propose an end-to-end trainable approach to single-channel speech separation with unknown number of speakers. Our approach extends the MulCat source separation backbone with additional output heads: a count-head to infer the number of speakers, and decoder-heads for reconstructing the original signals. Beyond the model, we also propose a metric on how to evaluate source separation with variable number of speakers. Specifically, we cleared up the issue on how to evaluate the quality when the ground-truth hasmore or less speakers than the ones predicted by the model. We evaluate our approach on the WSJ0-mix datasets, with mixtures up to five speakers. We demonstrate that our approach outperforms state-of-the-art in counting the number of speakers and remains competitive in quality of reconstructed signals.