Causal Speech Enhancement with Predicting Semantics based on Quantized Self-supervised Learning Features

TL;DR

This paper introduces a novel causal speech enhancement model that leverages self-supervised learning features and semantic token prediction to improve real-time speech quality, demonstrating significant PESQ improvements.

Contribution

It is the first to combine SSL features with causality in an SE model, incorporating semantic token prediction via multi-task learning for enhanced performance.

Findings

Achieved PESQ of 2.88 on VoiceBank + DEMAND dataset.

Semantic prediction significantly improves speech enhancement quality.

First integration of SSL features and causality in real-time SE.

Abstract

Real-time speech enhancement (SE) is essential to online speech communication. Causal SE models use only the previous context while predicting future information, such as phoneme continuation, may help performing causal SE. The phonetic information is often represented by quantizing latent features of self-supervised learning (SSL) models. This work is the first to incorporate SSL features with causality into an SE model. The causal SSL features are encoded and combined with spectrogram features using feature-wise linear modulation to estimate a mask for enhancing the noisy input speech. Simultaneously, we quantize the causal SSL features using vector quantization to represent phonetic characteristics as semantic tokens. The model not only encodes SSL features but also predicts the future semantic tokens in multi-task learning (MTL). The experimental results using VoiceBank + DEMAND dataset show that our proposed method achieves 2.88 in PESQ, especially with semantic prediction MTL, in which we confirm that the semantic prediction played an important role in causal SE.