The Costs of Reproducibility in Music Separation Research: a Replication of Band-Split RNN

TL;DR

This paper critically examines the reproducibility challenges of the Band-Split RNN model for music source separation, highlighting the importance of transparency, and provides an improved, publicly available implementation to foster more sustainable research practices.

Contribution

It replicates and refines the BSRNN model, revealing insights for future improvements and emphasizing the need for transparency in reproducibility.

Findings

Reproducing the original results was unsuccessful.

An optimized BSRNN model significantly outperforms the original.

Public release of code and models promotes reproducibility.

Abstract

Music source separation is the task of isolating the instrumental tracks from a music song. Despite its spectacular recent progress, the trend towards more complex architectures and training protocols exacerbates reproducibility issues. The band-split recurrent neural networks (BSRNN) model is promising in this regard, since it yields close to state-of-the-art results on public datasets, and requires reasonable resources for training. Unfortunately, it is not straightforward to reproduce since its full code is not available. In this paper, we attempt to replicate BSRNN as closely as possible to the original paper through extensive experiments, which allows us to conduct a critical reflection on this reproducibility issue. Our contributions are three-fold. First, this study yields several insights on the model design and training pipeline, which sheds light on potential future improvements. In particular, since we were unsuccessful in reproducing the original results, we explore additional variants that ultimately yield an optimized BSRNN model, whose performance largely improves that of the original. Second, we discuss reproducibility issues from both methodological and practical perspectives. We notably underline how substantial time and energy costs could have been saved upon availability of the full pipeline. Third, our code and pre-trained models are released publicly to foster reproducible research. We hope that this study will contribute to spread awareness on the importance of reproducible research in the music separation community, and help promoting more transparent and sustainable practices.