One-shot conditional audio filtering of arbitrary sounds

TL;DR

This paper introduces SoundFilter, a neural network that performs one-shot sound source separation based on a short sample, capable of generalizing to unseen sounds without class labels, and demonstrates strong results on multiple datasets.

Contribution

The paper presents a wave-to-wave neural network architecture with a learned conditioning encoder for one-shot arbitrary sound filtering, trained without sound class labels.

Findings

Achieves 9.6 dB SI-SDR improvement on FSD50k

Achieves 14.0 dB SI-SDR improvement on Librispeech

Learned representations cluster similar sounds without labels

Abstract

We consider the problem of separating a particular sound source from a single-channel mixture, based on only a short sample of the target source. Using SoundFilter, a wave-to-wave neural network architecture, we can train a model without using any sound class labels. Using a conditioning encoder model which is learned jointly with the source separation network, the trained model can be "configured" to filter arbitrary sound sources, even ones that it has not seen during training. Evaluated on the FSD50k dataset, our model obtains an SI-SDR improvement of 9.6 dB for mixtures of two sounds. When trained on Librispeech, our model achieves an SI-SDR improvement of 14.0 dB when separating one voice from a mixture of two speakers. Moreover, we show that the representation learned by the conditioning encoder clusters acoustically similar sounds together in the embedding space, even though it is trained without using any labels.