Single microphone speaker extraction using unified time-frequency Siamese-Unet

TL;DR

This paper introduces a unified time-frequency Siamese-Unet architecture for single microphone speaker extraction that outperforms state-of-the-art methods by leveraging both time and frequency domain information.

Contribution

The paper proposes a novel Siamese-Unet model combining time and frequency domain processing for speaker extraction, trained with SI-SDR loss for improved performance.

Findings

Outperforms state-of-the-art BSS methods

Easier to train compared to existing models

Achieves superior speaker extraction results

Abstract

In this paper we present a unified time-frequency method for speaker extraction in clean and noisy conditions. Given a mixed signal, along with a reference signal, the common approaches for extracting the desired speaker are either applied in the time-domain or in the frequency-domain. In our approach, we propose a Siamese-Unet architecture that uses both representations. The Siamese encoders are applied in the frequency-domain to infer the embedding of the noisy and reference spectra, respectively. The concatenated representations are then fed into the decoder to estimate the real and imaginary components of the desired speaker, which are then inverse-transformed to the time-domain. The model is trained with the Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) loss to exploit the time-domain information. The time-domain loss is also regularized with frequency-domain loss to preserve the speech patterns. Experimental results demonstrate that the unified approach is not only very easy to train, but also provides superior results as compared with state-of-the-art (SOTA) Blind Source Separation (BSS) methods, as well as commonly used speaker extraction approach.