In search of strong embedding extractors for speaker diarisation

TL;DR

This paper investigates the effectiveness of speaker embedding extractors for diarisation, highlighting challenges in evaluation and handling overlapped speech, and proposes data augmentation techniques to improve performance in realistic scenarios.

Contribution

The paper introduces evaluation protocols that better reflect diarisation conditions and proposes two data augmentation methods to enhance embedding extractors for overlapped speech and speaker changes.

Findings

Augmentation techniques improve diarisation performance.

Evaluation protocols better correlate with diarisation accuracy.

State-of-the-art extractors benefit from proposed methods.

Abstract

Speaker embedding extractors (EEs), which map input audio to a speaker discriminant latent space, are of paramount importance in speaker diarisation. However, there are several challenges when adopting EEs for diarisation, from which we tackle two key problems. First, the evaluation is not straightforward because the features required for better performance differ between speaker verification and diarisation. We show that better performance on widely adopted speaker verification evaluation protocols does not lead to better diarisation performance. Second, embedding extractors have not seen utterances in which multiple speakers exist. These inputs are inevitably present in speaker diarisation because of overlapped speech and speaker changes; they degrade the performance. To mitigate the first problem, we generate speaker verification evaluation protocols that mimic the diarisation scenario better. We propose two data augmentation techniques to alleviate the second problem, making embedding extractors aware of overlapped speech or speaker change input. One technique generates overlapped speech segments, and the other generates segments where two speakers utter sequentially. Extensive experimental results using three state-of-the-art speaker embedding extractors demonstrate that both proposed approaches are effective.