Speech Enhancement with Multi-granularity Vector Quantization

TL;DR

This paper explores the use of multi-granularity vector quantization at different layers of neural networks, combined with attention mechanisms, to improve speech enhancement performance by capturing both global and local speech features.

Contribution

It introduces a novel approach of applying VQ at multiple layers with various codebooks and integrates it with pre-trained models and attention for enhanced speech denoising.

Findings

Improved speech enhancement results on Valentini dataset.

Multi-granularity VQ effectively captures diverse speech features.

Pre-trained models significantly impact enhancement performance.

Abstract

With advances in deep learning, neural network based speech enhancement (SE) has developed rapidly in the last decade. Meanwhile, the self-supervised pre-trained model and vector quantization (VQ) have achieved excellent performance on many speech-related tasks, while they are less explored on SE. As it was shown in our previous work that utilizing a VQ module to discretize noisy speech representations is beneficial for speech denoising, in this work we therefore study the impact of using VQ at different layers with different number of codebooks. Different VQ modules indeed enable to extract multiple-granularity speech features. Following an attention mechanism, the contextual features extracted by a pre-trained model are fused with the local features extracted by the encoder, such that both global and local information are preserved to reconstruct the enhanced speech. Experimental results on the Valentini dataset show that the proposed model can improve the SE performance, where the impact of choosing pre-trained models is also revealed.