Speaker diarization using latent space clustering in generative adversarial network

TL;DR

This paper introduces a novel deep latent space clustering approach for speaker diarization using GANs, which significantly outperforms existing x-vector based systems across multiple datasets, including clinical and meeting scenarios.

Contribution

The work presents a joint training framework combining GANs, latent variable recovery, and clustering losses for improved speaker diarization, especially in challenging clinical data.

Findings

Significant DER reduction on AMI, ADOS, and BOSCC datasets.

Embedding fusion with x-vectors improves diarization accuracy.

Outperforms state-of-the-art x-vector based systems.

Abstract

In this work, we propose deep latent space clustering for speaker diarization using generative adversarial network (GAN) backprojection with the help of an encoder network. The proposed diarization system is trained jointly with GAN loss, latent variable recovery loss, and a clustering-specific loss. It uses x-vector speaker embeddings at the input, while the latent variables are sampled from a combination of continuous random variables and discrete one-hot encoded variables using the original speaker labels. We benchmark our proposed system on the AMI meeting corpus, and two child-clinician interaction corpora (ADOS and BOSCC) from the autism diagnosis domain. ADOS and BOSCC contain diagnostic and treatment outcome sessions respectively obtained in clinical settings for verbal children and adolescents with autism. Experimental results show that our proposed system significantly outperform the state-of-the-art x-vector based diarization system on these databases. Further, we perform embedding fusion with x-vectors to achieve a relative DER improvement of 31%, 36% and 49% on AMI eval, ADOS and BOSCC corpora respectively, when compared to the x-vector baseline using oracle speech segmentation.