Robust Target Speaker Diarization and Separation via Augmented Speaker Embedding Sampling

TL;DR

This paper presents a novel enrollment-free method for simultaneous speech separation and diarization that uses robust speaker embeddings and an overlapping spectral loss, significantly improving performance over existing methods.

Contribution

It introduces a dual-stage training pipeline with a new spectral loss function for better diarization and separation without prior speaker knowledge.

Findings

Achieves 71% relative improvement in DER

Achieves 69% relative improvement in cpWER

Demonstrates robustness to background noise

Abstract

Traditional speech separation and speaker diarization approaches rely on prior knowledge of target speakers or a predetermined number of participants in audio signals. To address these limitations, recent advances focus on developing enrollment-free methods capable of identifying targets without explicit speaker labeling. This work introduces a new approach to train simultaneous speech separation and diarization using automatic identification of target speaker embeddings, within mixtures. Our proposed model employs a dual-stage training pipeline designed to learn robust speaker representation features that are resilient to background noise interference. Furthermore, we present an overlapping spectral loss function specifically tailored for enhancing diarization accuracy during overlapped speech frames. Experimental results show significant performance gains compared to the current SOTA baseline, achieving 71% relative improvement in DER and 69% in cpWER.