Improving Target Sound Extraction with Timestamp Information

TL;DR

This paper introduces a novel approach to target sound extraction that leverages timestamp information through a mutual learning framework and a specialized loss function, significantly enhancing extraction performance.

Contribution

It proposes integrating timestamp data via a target sound detection network and a target-weighted loss, enabling mutual learning for improved sound extraction.

Findings

Significant performance improvements on Freesound datasets.

Effective utilization of timestamp information enhances extraction accuracy.

Mutual learning framework benefits both detection and extraction tasks.

Abstract

Target sound extraction (TSE) aims to extract the sound part of a target sound event class from a mixture audio with multiple sound events. The previous works mainly focus on the problems of weakly-labelled data, jointly learning and new classes, however, no one cares about the onset and offset times of the target sound event, which has been emphasized in the auditory scene analysis. In this paper, we study to utilize such timestamp information to help extract the target sound via a target sound detection network and a target-weighted time-frequency loss function. More specifically, we use the detection result of a target sound detection (TSD) network as the additional information to guide the learning of target sound extraction network. We also find that the result of TSE can further improve the performance of the TSD network, so that a mutual learning framework of the target sound detection and extraction is proposed. In addition, a target-weighted time-frequency loss function is designed to pay more attention to the temporal regions of the target sound during training. Experimental results on the synthesized data generated from the Freesound Datasets show that our proposed method can significantly improve the performance of TSE.