On Training Targets and Activation Functions for Deep Representation Learning in Text-Dependent Speaker Verification

TL;DR

This paper systematically evaluates the impact of training targets, activation functions, and loss functions on deep neural network performance in text-dependent speaker verification, highlighting the effectiveness of self-supervised methods and GELU activation.

Contribution

It provides a comprehensive analysis of training targets, loss functions, and activation functions, introducing the effectiveness of GELU and TCL in TD-SV systems.

Findings

GELU activation reduces error rates significantly.

Time-contrastive learning (TCL) outperforms other training targets.

Cross entropy, joint-softmax, and focal loss improve system accuracy.

Abstract

Deep representation learning has gained significant momentum in advancing text-dependent speaker verification (TD-SV) systems. When designing deep neural networks (DNN) for extracting bottleneck features, key considerations include training targets, activation functions, and loss functions. In this paper, we systematically study the impact of these choices on the performance of TD-SV. For training targets, we consider speaker identity, time-contrastive learning (TCL) and auto-regressive prediction coding with the first being supervised and the last two being self-supervised. Furthermore, we study a range of loss functions when speaker identity is used as the training target. With regard to activation functions, we study the widely used sigmoid function, rectified linear unit (ReLU), and Gaussian error linear unit (GELU). We experimentally show that GELU is able to reduce the error rates of TD-SV significantly compared to sigmoid, irrespective of training target. Among the three training targets, TCL performs the best. Among the various loss functions, cross entropy, joint-softmax and focal loss functions outperform the others. Finally, score-level fusion of different systems is also able to reduce the error rates. Experiments are conducted on the RedDots 2016 challenge database for TD-SV using short utterances. For the speaker classifications, the well-known Gaussian mixture model-universal background model (GMM-UBM) and i-vector techniques are used.