Is MixIT Really Unsuitable for Correlated Sources? Exploring MixIT for Unsupervised Pre-training in Music Source Separation

TL;DR

This paper explores the potential of MixIT, an unsupervised learning method, for pre-training in music source separation, showing it can improve performance despite initial assumptions about its limitations with correlated sources.

Contribution

The study demonstrates that MixIT can be effectively used for unsupervised pre-training in MSS, challenging the belief that it is unsuitable due to source correlation issues.

Findings

MixIT pre-training improves MSS performance over training from scratch.

Pre-training on unlabeled data with MixIT enhances fine-tuned MSS models.

MixIT can separate instruments to some extent despite source correlation challenges.

Abstract

In music source separation (MSS), obtaining isolated sources or stems is highly costly, making pre-training on unlabeled data a promising approach. Although source-agnostic unsupervised learning like mixture-invariant training (MixIT) has been explored in general sound separation, they have been largely overlooked in MSS due to its implicit assumption of source independence. We hypothesize, however, that the difficulty of applying MixIT to MSS arises from the ill-posed nature of MSS itself, where stem definitions are application-dependent and models lack explicit knowledge of what should or should not be separated, rather than from high inter-source correlation. While MixIT does not assume any source model and struggles with such ambiguities, our preliminary experiments show that it can still separate instruments to some extent, suggesting its potential for unsupervised pre-training. Motivated by these insights, this study investigates MixIT-based pre-training for MSS. We first pre-train a model on in-the-wild, unlabeled data from the Free Music Archive using MixIT, and then fine-tune it on MUSDB18 with supervision. Using the band-split TF-Locoformer, one of the state-of-the-art MSS models, we demonstrate that MixIT-based pre-training improves the performance over training from scratch.