Few-shot learning of new sound classes for target sound extraction

TL;DR

This paper introduces a combined approach for target sound extraction that leverages both class labels and enrollment audio, enabling effective extraction of both seen and unseen sound classes, with further improvements via few-shot adaptation.

Contribution

It proposes a novel framework combining 1-hot and enrollment-based extraction, and introduces few-shot adaptation for new sound classes.

Findings

Effective extraction of unseen sound classes demonstrated.

Combined framework outperforms traditional methods on synthesized mixtures.

Few-shot adaptation improves performance on new classes.

Abstract

Target sound extraction consists of extracting the sound of a target acoustic event (AE) class from a mixture of AE sounds. It can be realized using a neural network that extracts the target sound conditioned on a 1-hot vector that represents the desired AE class. With this approach, embedding vectors associated with the AE classes are directly optimized for the extraction of sound classes seen during training. However, it is not easy to extend this framework to new AE classes, i.e. unseen during training. Recently, speech, music, or AE sound extraction based on enrollment audio of the desired sound offers the potential of extracting any target sound in a mixture given only a short audio signal of a similar sound. In this work, we propose combining 1-hot- and enrollment-based target sound extraction, allowing optimal performance for seen AE classes and simple extension to new classes. In experiments with synthesized sound mixtures generated with the Freesound Dataset (FSD) datasets, we demonstrate the benefit of the combined framework for both seen and new AE classes. Besides, we also propose adapting the embedding vectors obtained from a few enrollment audio samples (few-shot) to further improve performance on new classes.