TutorNet: Towards Flexible Knowledge Distillation for End-to-End Speech Recognition

TL;DR

TutorNet introduces a flexible knowledge distillation approach for end-to-end speech recognition, enabling knowledge transfer across different network architectures and improving student model performance beyond the teacher in some cases.

Contribution

It proposes TutorNet, a novel KD method that transfers knowledge across different neural network types at multiple levels, enhancing flexibility and performance in speech recognition models.

Findings

Significantly improves word error rate (WER) on LibriSpeech dataset.

Allows student models to outperform their teachers in certain scenarios.

Enables knowledge transfer between networks with different topologies.

Abstract

In recent years, there has been a great deal of research in developing end-to-end speech recognition models, which enable simplifying the traditional pipeline and achieving promising results. Despite their remarkable performance improvements, end-to-end models typically require expensive computational cost to show successful performance. To reduce this computational burden, knowledge distillation (KD), which is a popular model compression method, has been used to transfer knowledge from a deep and complex model (teacher) to a shallower and simpler model (student). Previous KD approaches have commonly designed the architecture of the student model by reducing the width per layer or the number of layers of the teacher model. This structural reduction scheme might limit the flexibility of model selection since the student model structure should be similar to that of the given teacher. To cope with this limitation, we propose a new KD method for end-to-end speech recognition, namely TutorNet, that can transfer knowledge across different types of neural networks at the hidden representation-level as well as the output-level. For concrete realizations, we firstly apply representation-level knowledge distillation (RKD) during the initialization step, and then apply the softmax-level knowledge distillation (SKD) combined with the original task learning. When the student is trained with RKD, we make use of frame weighting that points out the frames to which the teacher model pays more attention. Through a number of experiments on LibriSpeech dataset, it is verified that the proposed method not only distills the knowledge between networks with different topologies but also significantly contributes to improving the word error rate (WER) performance of the distilled student. Interestingly, TutorNet allows the student model to surpass its teacher's performance in some particular cases.