Generation of Speaker Representations Using Heterogeneous Training Batch Assembly

TL;DR

This paper introduces a CNN-based speaker modeling approach that accounts for heterogeneity in training data, improving diarization performance by generating meaningful embeddings for multi-speaker segments.

Contribution

It proposes a novel training scheme with synthetic multi-speaker segments and soft labels, enhancing speaker representation quality for diarization tasks.

Findings

Outperforms baseline x-vector systems in diarization tasks

Achieves significant reductions in DER, JER, and WER on benchmark datasets

Demonstrates robustness with heterogeneous and overlapping speaker data

Abstract

In traditional speaker diarization systems, a well-trained speaker model is a key component to extract representations from consecutive and partially overlapping segments in a long speech session. To be more consistent with the back-end segmentation and clustering, we propose a new CNN-based speaker modeling scheme, which takes into account the heterogeneity of the speakers in each training segment and batch. We randomly and synthetically augment the training data into a set of segments, each of which contains more than one speaker and some overlapping parts. A soft label is imposed on each segment based on its speaker occupation ratio, and the standard cross entropy loss is implemented in model training. In this way, the speaker model should have the ability to generate a geometrically meaningful embedding for each multi-speaker segment. Experimental results show that our system is superior to the baseline system using x-vectors in two speaker diarization tasks. In the CALLHOME task trained on the NIST SRE and Switchboard datasets, our system achieves a relative reduction of 12.93% in DER. In Track 2 of CHiME-6, our system provides 13.24%, 12.60%, and 5.65% relative reductions in DER, JER, and WER, respectively.