Sparse, Efficient, and Semantic Mixture Invariant Training: Taming In-the-Wild Unsupervised Sound Separation

TL;DR

This paper introduces novel loss functions and an efficient training method for unsupervised sound separation, addressing over-separation and computational challenges in mixture invariant training, leading to improved separation performance on in-the-wild data.

Contribution

It proposes sparsity and covariance losses to reduce over-separation and an efficient approximation for large source numbers in MixIT, enhancing unsupervised sound separation.

Findings

Achieved over 13 dB SI-SNR improvement on FUSS test set.

Boosted single-source SI-SNR by over 17 dB.

Demonstrated effectiveness of proposed losses and efficient MixIT in real-world data.

Abstract

Supervised neural network training has led to significant progress on single-channel sound separation. This approach relies on ground truth isolated sources, which precludes scaling to widely available mixture data and limits progress on open-domain tasks. The recent mixture invariant training (MixIT) method enables training on in-the-wild data; however, it suffers from two outstanding problems. First, it produces models which tend to over-separate, producing more output sources than are present in the input. Second, the exponential computational complexity of the MixIT loss limits the number of feasible output sources. In this paper we address both issues. To combat over-separation we introduce new losses: sparsity losses that favor fewer output sources and a covariance loss that discourages correlated outputs. We also experiment with a semantic classification loss by predicting weak class labels for each mixture. To handle larger numbers of sources, we introduce an efficient approximation using a fast least-squares solution, projected onto the MixIT constraint set. Our experiments show that the proposed losses curtail over-separation and improve overall performance. The best performance is achieved using larger numbers of output sources, enabled by our efficient MixIT loss, combined with sparsity losses to prevent over-separation. On the FUSS test set, we achieve over 13 dB in multi-source SI-SNR improvement, while boosting single-source reconstruction SI-SNR by over 17 dB.