Spectral Clustering-aware Learning of Embeddings for Speaker Diarisation

TL;DR

This paper introduces SCALE, a novel training method for speaker embeddings that aligns training objectives with spectral clustering, significantly improving diarisation accuracy.

Contribution

SCALE incorporates an affinity matrix loss and hyper-parameters for spectral clustering into embedding training, addressing the mismatch issue.

Findings

Over 50% relative speaker error rate reduction with oracle segmentation

Over 30% relative diarisation error rate reduction with automatic segmentation

Effective integration of spectral clustering considerations into embedding learning

Abstract

In speaker diarisation, speaker embedding extraction models often suffer from the mismatch between their training loss functions and the speaker clustering method. In this paper, we propose the method of spectral clustering-aware learning of embeddings (SCALE) to address the mismatch. Specifically, besides an angular prototype cal (AP) loss, SCALE uses a novel affinity matrix loss which directly minimises the error between the affinity matrix estimated from speaker embeddings and the reference. SCALE also includes p-percentile thresholding and Gaussian blur as two important hyper-parameters for spectral clustering in training. Experiments on the AMI dataset showed that speaker embeddings obtained with SCALE achieved over 50% relative speaker error rate reductions using oracle segmentation, and over 30% relative diarisation error rate reductions using automatic segmentation when compared to a strong baseline with the AP-loss-based speaker embeddings.