Knowledge Distillation for Speech Denoising by Latent Representation Alignment with Cosine Distance

TL;DR

This paper introduces a novel knowledge distillation approach for speech denoising that leverages cosine similarity and autoencoder structures to improve student model performance, especially under mismatched conditions.

Contribution

It proposes a new KD method using cosine similarity and autoencoder principles to enhance speech denoising, addressing limitations of existing KD techniques.

Findings

The proposed method outperforms baseline in various mismatching scenarios.

Students retain more denoising capability under mismatched conditions.

The approach demonstrates improved performance over state-of-the-art methods.

Abstract

Speech denoising is a generally adopted and impactful task, appearing in many common and everyday-life use cases. Although there are very powerful methods published, most of those are too complex for deployment in everyday and low-resources computational environments, like hand-held devices, intelligent glasses, hearing aids, etc. Knowledge distillation (KD) is a prominent way for alleviating this complexity mismatch and is based on the transferring/distilling of knowledge from a pre-trained complex model, the teacher, to another less complex one, the student. Existing KD methods for speech denoising are based on processes that potentially hamper the KD by bounding the learning of the student to the distribution, information ordering, and feature dimensionality learned by the teacher. In this paper, we present and assess a method that tries to treat this issue, by exploiting the well-known denoising-autoencoder framework, the linear inverted bottlenecks, and the properties of the cosine similarity. We use a public dataset and conduct repeated experiments with different mismatching scenarios between the teacher and the student, reporting the mean and standard deviation of the metrics of our method and another, state-of-the-art method that is used as a baseline. Our results show that with the proposed method, the student can perform better and can also retain greater mismatching conditions compared to the teacher.