Comparison of Knowledge Distillation Methods for Low-complexity Multi-microphone Speech Enhancement using the FT-JNF Architecture

TL;DR

This paper evaluates knowledge distillation techniques to create low-complexity, high-performance multi-microphone speech enhancement models based on the FT-JNF architecture, enabling deployment on resource-constrained devices.

Contribution

It systematically compares five KD methods for the FT-JNF architecture, demonstrating significant size reduction with minimal performance loss in speech enhancement.

Findings

Three KD methods improve student model performance over baseline.

A student model with 25% of teacher parameters achieves comparable PESQ scores.

Model size can be reduced by up to 96% with minimal PESQ score decrease.

Abstract

Multi-microphone speech enhancement using deep neural networks (DNNs) has significantly progressed in recent years. However, many proposed DNN-based speech enhancement algorithms cannot be implemented on devices with limited hardware resources. Only lowering the complexity of such systems by reducing the number of parameters often results in worse performance. Knowledge Distillation (KD) is a promising approach for reducing DNN model size while preserving performance. In this paper, we consider the recently proposed Frequency-Time Joint Non-linear Filter (FT-JNF) architecture and investigate several KD methods to train smaller (student) models from a large pre-trained (teacher) model. Five KD methods are evaluated using direct output matching, the self-similarity of intermediate layers, and fused multi-layer losses. Experimental results on a simulated dataset using a compact array with five microphones show that three KD methods substantially improve the performance of student models compared to training without KD. A student model with only 25% of the teacher model's parameters achieves comparable PESQ scores at 0 dB SNR. Furthermore, a reduction of up to 96% in model size can be achieved with only a minimal decrease in PESQ scores.