Distil-DCCRN: A Small-footprint DCCRN Leveraging Feature-based Knowledge Distillation in Speech Enhancement

TL;DR

This paper introduces Distil-DCCRN, a compact speech enhancement model that uses a novel knowledge distillation method with attention transfer and KL divergence, achieving performance comparable to larger models.

Contribution

The paper presents a new knowledge distillation approach that transfers both output and intermediate features from a complex teacher model to a smaller student model in speech enhancement.

Findings

Distil-DCCRN has only 30% of DCCRN's parameters.

Distil-DCCRN outperforms DCCRN in PESQ and SI-SNR metrics.

The method achieves comparable DNSMOS results to DCCRN.

Abstract

The deep complex convolution recurrent network (DCCRN) achieves excellent speech enhancement performance by utilizing the audio spectrum's complex features. However, it has a large number of model parameters. We propose a smaller model, Distil-DCCRN, which has only 30% of the parameters compared to the DCCRN. To ensure that the performance of Distil-DCCRN matches that of the DCCRN, we employ the knowledge distillation (KD) method to use a larger teacher model to help train a smaller student model. We design a knowledge distillation (KD) method, integrating attention transfer and Kullback-Leibler divergence (AT-KL) to train the student model Distil-DCCRN. Additionally, we use a model with better performance and a more complicated structure, Uformer, as the teacher model. Unlike previous KD approaches that mainly focus on model outputs, our method also leverages the intermediate features from the models' middle layers, facilitating rich knowledge transfer across different structured models despite variations in layer configurations and discrepancies in the channel and time dimensions of intermediate features. Employing our AT-KL approach, Distil-DCCRN outperforms DCCRN as well as several other competitive models in both PESQ and SI-SNR metrics on the DNS test set and achieves comparable results to DCCRN in DNSMOS.