FlowSep: Language-Queried Sound Separation with Rectified Flow Matching

TL;DR

FlowSep introduces a novel generative approach based on rectified flow matching for language-queried sound separation, achieving superior quality and efficiency over existing models by leveraging a VAE latent space and a pre-trained vocoder.

Contribution

This work pioneers the application of rectified flow matching in sound separation, demonstrating its advantages over discriminative and diffusion-based models in LASS tasks.

Findings

FlowSep outperforms state-of-the-art models on multiple benchmarks.

FlowSep surpasses diffusion-based models in separation quality and inference speed.

The model effectively learns linear flow trajectories in VAE latent space for sound separation.

Abstract

Language-queried audio source separation (LASS) focuses on separating sounds using textual descriptions of the desired sources. Current methods mainly use discriminative approaches, such as time-frequency masking, to separate target sounds and minimize interference from other sources. However, these models face challenges when separating overlapping soundtracks, which may lead to artifacts such as spectral holes or incomplete separation. Rectified flow matching (RFM), a generative model that establishes linear relations between the distribution of data and noise, offers superior theoretical properties and simplicity, but has not yet been explored in sound separation. In this work, we introduce FlowSep, a new generative model based on RFM for LASS tasks. FlowSep learns linear flow trajectories from noise to target source features within the variational autoencoder (VAE) latent space. During inference, the RFM-generated latent features are reconstructed into a mel-spectrogram via the pre-trained VAE decoder, followed by a pre-trained vocoder to synthesize the waveform. Trained on 1,680 hours of audio data, FlowSep outperforms the state-of-the-art models across multiple benchmarks, as evaluated with subjective and objective metrics. Additionally, our results show that FlowSep surpasses a diffusion-based LASS model in both separation quality and inference efficiency, highlighting its strong potential for audio source separation tasks. Code, pre-trained models and demos can be found at: https://audio-agi.github.io/FlowSep_demo/ .