Speech Enhancement Using Self-Supervised Pre-Trained Model and Vector Quantization

TL;DR

This paper introduces a speech enhancement method leveraging self-supervised pre-trained models and vector quantization, improving real-time denoising performance by discretizing speech representations and adapting model architecture.

Contribution

It applies self-supervised pre-trained WavLM to initialize a modified DEMUCS model with causal convolutions and attention, incorporating vector quantization for enhanced denoising.

Findings

Pre-trained model initialization improves speech enhancement performance.

Vector quantization suppresses noise in speech representations.

Method outperforms baseline models on Valentini and internal datasets.

Abstract

With the development of deep learning, neural network-based speech enhancement (SE) models have shown excellent performance. Meanwhile, it was shown that the development of self-supervised pre-trained models can be applied to various downstream tasks. In this paper, we will consider the application of the pre-trained model to the real-time SE problem. Specifically, the encoder and bottleneck layer of the DEMUCS model are initialized using the self-supervised pretrained WavLM model, the convolution in the encoder is replaced by causal convolution, and the transformer encoder in the bottleneck layer is based on causal attention mask. In addition, as discretizing the noisy speech representations is more beneficial for denoising, we utilize a quantization module to discretize the representation output from the bottleneck layer, which is then fed into the decoder to reconstruct the clean speech waveform. Experimental results on the Valentini dataset and an internal dataset show that the pre-trained model based initialization can improve the SE performance and the discretization operation suppresses the noise component in the representations to some extent, which can further improve the performance.