Integrated Multi-Level Knowledge Distillation for Enhanced Speaker Verification

TL;DR

This paper introduces IML-KD, a novel knowledge distillation method that transfers multi-scale temporal speech features from teacher to student models, significantly improving speaker verification accuracy.

Contribution

The paper proposes a new multi-level KD approach that incorporates temporal context and gradient-based representations to better capture speech dynamics.

Findings

IML-KD reduces EER by 5% on VoxCeleb1.

The method outperforms existing KD approaches in speaker verification.

Enhanced transfer of temporal speech features improves model performance.

Abstract

Knowledge distillation (KD) is widely used in audio tasks, such as speaker verification (SV), by transferring knowledge from a well-trained large model (the teacher) to a smaller, more compact model (the student) for efficiency and portability. Existing KD methods for SV often mirror those used in image processing, focusing on approximating predicted probabilities and hidden representations. However, these methods fail to account for the multi-level temporal properties of speech audio. In this paper, we propose a novel KD method, i.e., Integrated Multi-level Knowledge Distillation (IML-KD), to transfer knowledge of various temporal-scale features of speech from a teacher model to a student model. In the IML-KD, temporal context information from the teacher model is integrated into novel Integrated Gradient-based input-sensitive representations from speech segments with various durations, and the student model is trained to infer these representations with multi-level alignment for the output. We conduct SV experiments on the VoxCeleb1 dataset to evaluate the proposed method. Experimental results demonstrate that IML-KD significantly enhances KD performance, reducing the Equal Error Rate (EER) by 5%.