Two-Step Sound Source Separation: Training on Learned Latent Targets

TL;DR

This paper introduces a novel two-step training approach for sound source separation, where a latent space transform is learned first, followed by training a separation module, resulting in improved performance over joint learning methods.

Contribution

The paper proposes a two-step training procedure involving latent space transform learning and separation module training, with a SI-SDR loss that lower-bounds time-domain SI-SDR, enhancing separation performance.

Findings

Outperforms joint learning systems in sound separation tasks

The latent space transform improves separation quality

The SI-SDR loss in latent space effectively correlates with time-domain SI-SDR

Abstract

In this paper, we propose a two-step training procedure for source separation via a deep neural network. In the first step we learn a transform (and it's inverse) to a latent space where masking-based separation performance using oracles is optimal. For the second step, we train a separation module that operates on the previously learned space. In order to do so, we also make use of a scale-invariant signal to distortion ratio (SI-SDR) loss function that works in the latent space, and we prove that it lower-bounds the SI-SDR in the time domain. We run various sound separation experiments that show how this approach can obtain better performance as compared to systems that learn the transform and the separation module jointly. The proposed methodology is general enough to be applicable to a large class of neural network end-to-end separation systems.