Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model

TL;DR

This paper introduces a latent diffusion model for singing voice separation that achieves faster inference and improved separation quality using only paired data, advancing generative approaches in music source separation.

Contribution

It presents a novel latent diffusion-based system for singing voice separation that outperforms existing generative models and matches non-generative systems in quality, with efficient training and inference.

Findings

Outperforms existing generative separation systems.

Levels with non-generative systems on quality measures.

Provides insights into noise robustness of the latent encoder.

Abstract

Extracting individual elements from music mixtures is a valuable tool for music production and practice. While neural networks optimized to mask or transform mixture spectrograms into the individual source(s) have been the leading approach, the source overlap and correlation in music signals poses an inherent challenge. Also, accessing all sources in the mixture is crucial to train these systems, while complicated. Attempts to address these challenges in a generative fashion exist, however, the separation performance and inference efficiency remain limited. In this work, we study the potential of diffusion models to advance toward bridging this gap, focusing on generative singing voice separation relying only on corresponding pairs of isolated vocals and mixtures for training. To align with creative workflows, we leverage latent diffusion: the system generates samples encoded in a compact latent space, and subsequently decodes these into audio. This enables efficient optimization and faster inference. Our system is trained using only open data. We outperform existing generative separation systems, and level the compared non-generative systems on a list of signal quality measures and on interference removal. We provide a noise robustness study on the latent encoder, providing insights on its potential for the task. We release a modular toolkit for further research on the topic.