UniFlow: Unifying Speech Front-End Tasks via Continuous Generative Modeling

TL;DR

UniFlow introduces a unified generative modeling framework for various speech front-end tasks, leveraging a shared latent space and conditional training to improve performance and extensibility across multiple benchmarks.

Contribution

It presents a novel unified approach using continuous generative modeling with a waveform VAE and Diffusion Transformer for diverse speech tasks in a shared latent space.

Findings

Consistent performance improvements over state-of-the-art baselines.

Effective task differentiation using learnable condition embeddings.

Demonstrated extensibility to new speech processing tasks.

Abstract

Generative modeling has recently achieved remarkable success across image, video, and audio domains, demonstrating powerful capabilities for unified representation learning. Yet speech front-end tasks such as speech enhancement (SE), target speaker extraction (TSE), acoustic echo cancellation (AEC), and language-queried source separation (LASS) remain largely tackled by disparate, task-specific solutions. This fragmentation leads to redundant engineering effort, inconsistent performance, and limited extensibility. To address this gap, we introduce UniFlow, a unified framework that employs continuous generative modeling to tackle diverse speech front-end tasks in a shared latent space. Specifically, UniFlow utilizes a waveform variational autoencoder (VAE) to learn a compact latent representation of raw audio, coupled with a Diffusion Transformer (DiT) that predicts latent updates. To differentiate the speech processing task during the training, learnable condition embeddings indexed by a task ID are employed to enable maximal parameter sharing while preserving task-specific adaptability. To balance model performance and computational efficiency, we investigate and compare three generative objectives: denoising diffusion, flow matching, and mean flow within the latent domain. We validate UniFlow on multiple public benchmarks, demonstrating consistent gains over state-of-the-art baselines. UniFlow's unified latent formulation and conditional design make it readily extensible to new tasks, providing an integrated foundation for building and scaling generative speech processing pipelines. To foster future research, we will open-source our codebase.