Incorporating Symbolic Sequential Modeling for Speech Enhancement

TL;DR

This paper introduces a speech enhancement method that integrates symbolic linguistic modeling with acoustic processing, using vector quantization to improve noise suppression and speech intelligibility in noisy environments.

Contribution

It proposes a novel framework combining symbolic sequential modeling with traditional acoustic models, leveraging vector quantized representations to incorporate linguistic constraints into speech enhancement.

Findings

Significant improvements in PESQ scores.

Enhanced speech intelligibility as measured by STOI.

Effective use of symbolic representations for noise reduction.

Abstract

In a noisy environment, a lossy speech signal can be automatically restored by a listener if he/she knows the language well. That is, with the built-in knowledge of a "language model", a listener may effectively suppress noise interference and retrieve the target speech signals. Accordingly, we argue that familiarity with the underlying linguistic content of spoken utterances benefits speech enhancement (SE) in noisy environments. In this study, in addition to the conventional modeling for learning the acoustic noisy-clean speech mapping, an abstract symbolic sequential modeling is incorporated into the SE framework. This symbolic sequential modeling can be regarded as a "linguistic constraint" in learning the acoustic noisy-clean speech mapping function. In this study, the symbolic sequences for acoustic signals are obtained as discrete representations with a Vector Quantized Variational Autoencoder algorithm. The obtained symbols are able to capture high-level phoneme-like content from speech signals. The experimental results demonstrate that the proposed framework can obtain notable performance improvement in terms of perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI) on the TIMIT dataset.