Interrupted and cascaded permutation invariant training for speech separation

TL;DR

This paper explores alternative label assignment strategies for speech separation training, demonstrating that fixed label assignments combined with cascaded PIT significantly improve performance without altering the model architecture.

Contribution

It introduces a novel training approach using fixed label assignments and cascaded PIT, achieving state-of-the-art results in speech separation.

Findings

Fixed label assignments outperform traditional PIT.

Cascaded PIT with fixed labels achieves state-of-the-art performance.

Training with fixed labels is more effective than dynamic label assignment.

Abstract

Permutation Invariant Training (PIT) has long been a stepping stone method for training speech separation model in handling the label ambiguity problem. With PIT selecting the minimum cost label assignments dynamically, very few studies considered the separation problem to be optimizing both the model parameters and the label assignments, but focused on searching for good model architecture and parameters. In this paper, we investigate instead for a given model architecture the various flexible label assignment strategies for training the model, rather than directly using PIT. Surprisingly, we discover a significant performance boost compared to PIT is possible if the model is trained with fixed label assignments and a good set of labels is chosen. With fixed label training cascaded between two sections of PIT, we achieved the state-of-the-art performance on WSJ0-2mix without changing the model architecture at all.