Semantic Grouping Network for Audio Source Separation

TL;DR

This paper introduces SGN, a novel audio-only model that directly disentangles and extracts high-level semantic features for each source in an audio mixture, outperforming previous methods.

Contribution

SGN is the first to directly learn semantic disentanglement from audio mixtures using learnable class tokens, without relying on visual information.

Findings

SGN outperforms previous audio-only separation methods.

SGN surpasses audio-visual models without visual cues.

Extensive experiments on multiple benchmarks validate SGN's effectiveness.

Abstract

Recently, audio-visual separation approaches have taken advantage of the natural synchronization between the two modalities to boost audio source separation performance. They extracted high-level semantics from visual inputs as the guidance to help disentangle sound representation for individual sources. Can we directly learn to disentangle the individual semantics from the sound itself? The dilemma is that multiple sound sources are mixed together in the original space. To tackle the difficulty, in this paper, we present a novel Semantic Grouping Network, termed as SGN, that can directly disentangle sound representations and extract high-level semantic information for each source from input audio mixture. Specifically, SGN aggregates category-wise source features through learnable class tokens of sounds. Then, the aggregated semantic features can be used as the guidance to separate the corresponding audio sources from the mixture. We conducted extensive experiments on music-only and universal sound separation benchmarks: MUSIC, FUSS, MUSDB18, and VGG-Sound. The results demonstrate that our SGN significantly outperforms previous audio-only methods and audio-visual models without utilizing additional visual cues.