Can We Really Repurpose Multi-Speaker ASR Corpus for Speaker Diarization?

TL;DR

This paper investigates the impact of boundary looseness in multi-speaker ASR datasets on neural speaker diarization performance, highlighting issues with dataset consistency and proposing standardized boundary alignment to improve results.

Contribution

It demonstrates that boundary looseness in ASR datasets hampers diarization accuracy and shows that using standardized tight boundaries enhances both diarization and ASR performance.

Findings

Looseness of segment boundaries reduces diarization accuracy.

Models trained on loose boundaries do not generalize well to other datasets.

Standardized boundary alignment improves diarization and ASR performance.

Abstract

Neural speaker diarization is widely used for overlap-aware speaker diarization, but it requires large multi-speaker datasets for training. To meet this data requirement, large datasets are often constructed by combining multiple corpora, including those originally designed for multi-speaker automatic speech recognition (ASR). However, ASR datasets often feature loosely defined segment boundaries that do not align with the stricter conventions of diarization benchmarks. In this work, we show that such boundary looseness significantly impacts the diarization error rate, reducing evaluation reliability. We also reveal that models trained on data with varying boundary precision tend to learn dataset-specific looseness, leading to poor generalization across out-of-domain datasets. Training with standardized tight boundaries via forced alignment improves not only diarization performance, especially in streaming scenarios, but also ASR performance when combined with simple post-processing.