USE: A Unified Model for Universal Sound Separation and Extraction

TL;DR

This paper introduces a unified model that combines sound separation and target sound extraction, automatically inferring source counts and interpreting user clues to improve performance in complex acoustic scenarios.

Contribution

It proposes a novel architecture that jointly handles sound separation and extraction, overcoming limitations of existing methods through a unified latent space and adaptive inference.

Findings

1.4 dB SDR improvement in sound separation over baselines

86% accuracy in target sound extraction

Effective handling of unknown source counts and diverse clues

Abstract

Sound separation (SS) and target sound extraction (TSE) are fundamental techniques for addressing complex acoustic scenarios. While existing SS methods struggle with determining the unknown number of sound sources, TSE approaches require precisely specified clues to achieve optimal performance. This paper proposes a unified framework that synergistically combines SS and TSE to overcome their individual limitations. Our architecture employs two complementary components: 1) An Encoder-Decoder Attractor (EDA) network that automatically infers both the source count and corresponding acoustic clues for SS, and 2) A multi-modal fusion network that precisely interprets diverse user-provided clues (acoustic, semantic, or visual) for TSE. Through joint training with cross-task consistency constraints, we establish a unified latent space that bridges both paradigms. During inference, the system adaptively operates in either fully autonomous SS mode or clue-driven TSE mode. Experiments demonstrate remarkable performance in both tasks, with notable improvements of 1.4 dB SDR improvement in SS compared to baseline and 86\% TSE accuracy.