Dissecting the Segmentation Model of End-to-End Diarization with Vector Clustering

TL;DR

This paper thoroughly analyzes how different architecture choices affect the performance of end-to-end neural speaker diarization with vector clustering, identifying optimal configurations and achieving state-of-the-art results.

Contribution

It provides a comprehensive evaluation of encoder, decoder, loss, and chunk size choices, revealing their impacts and best practices for diarization pipelines.

Findings

Finetuned WavLM encoders outperform others.

Conformer decoders yield the best performance.

Multiclass loss generally improves accuracy.

Abstract

End-to-End Neural Diarization with Vector Clustering is a powerful and practical approach to perform Speaker Diarization. Multiple enhancements have been proposed for the segmentation model of these pipelines, but their synergy had not been thoroughly evaluated. In this work, we provide an in-depth analysis on the impact of major architecture choices on the performance of the pipeline. We investigate different encoders (SincNet, pretrained and finetuned WavLM), different decoders (LSTM, Mamba, and Conformer), different losses (multilabel and multiclass powerset), and different chunk sizes. Through in-depth experiments covering nine datasets, we found that the finetuned WavLM-based encoder always results in the best systems by a wide margin. The LSTM decoder is outclassed by Mamba- and Conformer-based decoders, and while we found Mamba more robust to other architecture choices, it is slightly inferior to our best architecture, which uses a Conformer encoder. We found that multilabel and multiclass powerset losses do not have the same distribution of errors. We confirmed that the multiclass loss helps almost all models attain superior performance, except when finetuning WavLM, in which case, multilabel is the superior choice. We also evaluated the impact of the chunk size on all aforementioned architecture choices and found that newer architectures tend to better handle long chunk sizes, which can greatly improve pipeline performance. Our best system achieved state-of-the-art results on five widely used speaker diarization datasets.