DIVE: End-to-end Speech Diarization via Iterative Speaker Embedding

TL;DR

DIVE is an innovative end-to-end neural speaker diarization system that iteratively refines speaker representations, eliminating the need for pretrained embeddings and achieving state-of-the-art results on the CALLHOME benchmark.

Contribution

It introduces an iterative speaker embedding approach that resolves speaker ordering without permutation invariant training and does not rely on pretrained speaker models.

Findings

Achieves 6.7% DER on CALLHOME, outperforming previous methods.

Does not require pretrained speaker representations.

Optimizes all parameters with a multi-speaker voice activity loss.

Abstract

We introduce DIVE, an end-to-end speaker diarization algorithm. Our neural algorithm presents the diarization task as an iterative process: it repeatedly builds a representation for each speaker before predicting the voice activity of each speaker conditioned on the extracted representations. This strategy intrinsically resolves the speaker ordering ambiguity without requiring the classical permutation invariant training loss. In contrast with prior work, our model does not rely on pretrained speaker representations and optimizes all parameters of the system with a multi-speaker voice activity loss. Importantly, our loss explicitly excludes unreliable speaker turn boundaries from training, which is adapted to the standard collar-based Diarization Error Rate (DER) evaluation. Overall, these contributions yield a system redefining the state-of-the-art on the standard CALLHOME benchmark, with 6.7% DER compared to 7.8% for the best alternative.