Remastering Divide and Remaster: A Cinematic Audio Source Separation Dataset with Multilingual Support

TL;DR

This paper introduces DnR v3, an improved multilingual cinematic audio source separation dataset, enhancing diversity and quality, and demonstrates that multilingual training improves model generalization across languages.

Contribution

The authors develop DnR v3, a significantly improved, multilingual dataset for cinematic audio source separation, and show that multilingual training enhances model performance across languages.

Findings

Multilingual training improves model generalizability across languages.

Models trained on multilingual data perform as well or better than monolingual models.

DnR v3 includes diverse languages and addresses previous dataset limitations.

Abstract

Cinematic audio source separation (CASS), as a problem of extracting the dialogue, music, and effects stems from their mixture, is a relatively new subtask of audio source separation. To date, only one publicly available dataset exists for CASS, that is, the Divide and Remaster (DnR) dataset, which is currently at version 2. While DnR v2 has been an incredibly useful resource for CASS, several areas of improvement have been identified, particularly through its use in the 2023 Sound Demixing Challenge. In this work, we develop version 3 of the DnR dataset, addressing issues relating to vocal content in non-dialogue stems, loudness distributions, mastering process, and linguistic diversity. In particular, the dialogue stem of DnR v3 includes speech content from more than 30 languages from multiple families including but not limited to the Germanic, Romance, Indo-Aryan, Dravidian, Malayo-Polynesian, and Bantu families. Benchmark results using the Bandit model indicated that training on multilingual data yields significant generalizability to the model even in languages with low data availability. Even in languages with high data availability, the multilingual model often performs on par or better than dedicated models trained on monolingual CASS datasets. Dataset and model implementation will be made available at https://github.com/kwatcharasupat/source-separation-landing.