Deep Self-Supervised Hierarchical Clustering for Speaker Diarization

TL;DR

This paper introduces a self-supervised hierarchical clustering method that jointly learns speaker representations and clusters, significantly improving diarization accuracy over traditional methods.

Contribution

It presents a novel joint clustering and representation learning algorithm based on self-supervision, enhancing speaker diarization performance.

Findings

29% relative improvement over AHC with cosine similarity

10% relative improvement over state-of-the-art PLDA-based system

Effective integration of clustering with representation learning

Abstract

The state-of-the-art speaker diarization systems use agglomerative hierarchical clustering (AHC) which performs the clustering of previously learned neural embeddings. While the clustering approach attempts to identify speaker clusters, the AHC algorithm does not involve any further learning. In this paper, we propose a novel algorithm for hierarchical clustering which combines the speaker clustering along with a representation learning framework. The proposed approach is based on principles of self-supervised learning where the self-supervision is derived from the clustering algorithm. The representation learning network is trained with a regularized triplet loss using the clustering solution at the current step while the clustering algorithm uses the deep embeddings from the representation learning step. By combining the self-supervision based representation learning along with the clustering algorithm, we show that the proposed algorithm improves significantly 29% relative improvement) over the AHC algorithm with cosine similarity for a speaker diarization task on CALLHOME dataset. In addition, the proposed approach also improves over the state-of-the-art system with PLDA affinity matrix with 10% relative improvement in DER.