FastWhisper: Adaptive Self-knowledge Distillation for Real-time Automatic Speech Recognition

TL;DR

FastWhisper introduces an adaptive self-knowledge distillation method to enhance model generalization and significantly accelerate inference in real-time speech recognition, distilling the Whisper model into a smaller, more efficient variant.

Contribution

The paper proposes a novel adaptive self-knowledge distillation approach and applies it to create FastWhisper, a faster, smaller model with improved accuracy for speech recognition.

Findings

FastWhisper achieves 1.07% lower WER than Whisper.

Inference speed is increased by 5 times.

The method enhances generalization in speech recognition models.

Abstract

Knowledge distillation is one of the most effective methods for model compression. Previous studies have focused on the student model effectively training the predictive distribution of the teacher model. However, during training, the student model may inherit the shortcomings of the teacher model, which can lead to a decline in generalization capacity. To mitigate this issue, we propose adaptive self-knowledge distillation (ASKD), which dynamically reduces the dependence of the teacher model to improve the self-training capacity, and performs the self-knowledge distillation method to improve the generalization capacity of the student model. We further distill the Whisper model into a smaller variant, called FastWhisper. In our post-training setting, FastWhisper achieved a word error rate of 1.07% lower than the teacher model Whisper, and its relative inference time was 5 times faster.