Progressive Learning for Stabilizing Label Selection in Speech Separation with Mapping-based Method

TL;DR

This paper introduces a progressive learning strategy to stabilize label selection in deep speech separation models using a mapping-based method, leading to improved performance and reduced label switching issues.

Contribution

It proposes a novel progressive learning approach to address label switching in deep models with mapping-based speech separation, enhancing stability and accuracy.

Findings

Mapping-based method outperforms masking-based in large training sets.

Progressive learning reduces label switching without extra computational cost.

Enhanced separation performance with combined approach.

Abstract

Speech separation has been studied in time domain because of lower latency and higher performance compared to time-frequency domain. The masking-based method has been mostly used in time domain, and the other common method (mapping-based) has been inadequately studied. We investigate the use of the mapping-based method in the time domain and show that it can perform better on a large training set than the masking-based method. We also investigate the frequent label-switching problem in permutation invariant training (PIT), which results in suboptimal training because the labels selected by PIT differ across training epochs. Our experiment results showed that PIT works well in a shallow separation model, and the label switching occurs for a deeper model. We inferred that layer decoupling may be the reason for the frequent label switching. Therefore, we propose a training strategy based on progressive learning. This approach significantly reduced inconsistent label assignment without added computational complexity or training corpus. By combining this training strategy with the mapping-based method, we significantly improved the separation performance compared to the baseline.