Fast Real-time Personalized Speech Enhancement: End-to-End Enhancement Network (E3Net) and Knowledge Distillation

TL;DR

This paper introduces E3Net, a fast end-to-end speech enhancement model, and employs knowledge distillation and multi-task learning to create smaller, faster models that maintain high speech and transcription quality.

Contribution

The paper presents a novel end-to-end architecture (E3Net) for real-time personalized speech enhancement and demonstrates effective knowledge distillation techniques to produce compact, efficient models.

Findings

E3Net is 3 times faster than baseline models.

Knowledge distillation produces models 2-4 times faster with comparable quality.

Combining KD and MTL improves ASR and TSOS metrics without quality loss.

Abstract

This paper investigates how to improve the runtime speed of personalized speech enhancement (PSE) networks while maintaining the model quality. Our approach includes two aspects: architecture and knowledge distillation (KD). We propose an end-to-end enhancement (E3Net) model architecture, which is $3\times$ faster than a baseline STFT-based model. Besides, we use KD techniques to develop compressed student models without significantly degrading quality. In addition, we investigate using noisy data without reference clean signals for training the student models, where we combine KD with multi-task learning (MTL) using automatic speech recognition (ASR) loss. Our results show that E3Net provides better speech and transcription quality with a lower target speaker over-suppression (TSOS) rate than the baseline model. Furthermore, we show that the KD methods can yield student models that are $2-4\times$ faster than the teacher and provides reasonable quality. Combining KD and MTL improves the ASR and TSOS metrics without degrading the speech quality.