On real-time multi-stage speech enhancement systems

TL;DR

This paper presents a lightweight two-stage speech enhancement system with 560k parameters, achieving high performance comparable to larger models, suitable for real-time applications on low-power devices.

Contribution

It introduces a novel two-stage network with detailed analysis of influencing factors and training schemes, enabling efficient speech enhancement with significantly fewer parameters.

Findings

Achieves performance similar to larger models like DeepFilterNet2

Uses only 560k parameters in the proposed two-stage network

Provides insights into training schemes and model components

Abstract

Recently, multi-stage systems have stood out among deep learning-based speech enhancement methods. However, these systems are always high in complexity, requiring millions of parameters and powerful computational resources, which limits their application for real-time processing in low-power devices. Besides, the contribution of various influencing factors to the success of multi-stage systems remains unclear, which presents challenges to reduce the size of these systems. In this paper, we extensively investigate a lightweight two-stage network with only 560k total parameters. It consists of a Mel-scale magnitude masking model in the first stage and a complex spectrum mapping model in the second stage. We first provide a consolidated view of the roles of gain power factor, post-filter, and training labels for the Mel-scale masking model. Then, we explore several training schemes for the two-stage network and provide some insights into the superiority of the two-stage network. We show that the proposed two-stage network trained by an optimal scheme achieves a performance similar to a four times larger open source model DeepFilterNet2.