Class Token and Knowledge Distillation for Multi-head Self-Attention Speaker Verification Systems

TL;DR

This paper introduces novel methods for speaker verification using deep neural networks with multi-head self-attention, including a learnable class token, Bayesian estimation, and knowledge distillation, leading to improved robustness and performance.

Contribution

It proposes a learnable class token and a knowledge distillation framework for multi-head self-attention based speaker verification, enhancing temporal structure modeling and robustness.

Findings

Competitive results on RSR2015-Part II and DeepMine-Part 1 datasets.

Improved robustness with Bayesian estimation and distillation tokens.

Outperforms average pooling in embedding extraction.

Abstract

This paper explores three novel approaches to improve the performance of speaker verification (SV) systems based on deep neural networks (DNN) using Multi-head Self-Attention (MSA) mechanisms and memory layers. Firstly, we propose the use of a learnable vector called Class token to replace the average global pooling mechanism to extract the embeddings. Unlike global average pooling, our proposal takes into account the temporal structure of the input what is relevant for the text-dependent SV task. The class token is concatenated to the input before the first MSA layer, and its state at the output is used to predict the classes. To gain additional robustness, we introduce two approaches. First, we have developed a Bayesian estimation of the class token. Second, we have added a distilled representation token for training a teacher-student pair of networks using the Knowledge Distillation (KD) philosophy, which is combined with the class token. This distillation token is trained to mimic the predictions from the teacher network, while the class token replicates the true label. All the strategies have been tested on the RSR2015-Part II and DeepMine-Part 1 databases for text-dependent SV, providing competitive results compared to the same architecture using the average pooling mechanism to extract average embeddings.