GenSE: Generative Speech Enhancement via Language Models using Hierarchical Modeling

TL;DR

GenSE introduces a hierarchical, language model-based framework for speech enhancement that leverages semantic and acoustic tokens to improve speech quality and robustness in noisy environments.

Contribution

It proposes a novel hierarchical modeling approach that decouples semantic and acoustic token generation, enhancing stability and timbre consistency in speech enhancement.

Findings

Outperforms state-of-the-art systems in speech quality

Demonstrates strong generalization on benchmark datasets

Utilizes semantic tokens for improved intelligibility

Abstract

Semantic information refers to the meaning conveyed through words, phrases, and contextual relationships within a given linguistic structure. Humans can leverage semantic information, such as familiar linguistic patterns and contextual cues, to reconstruct incomplete or masked speech signals in noisy environments. However, existing speech enhancement (SE) approaches often overlook the rich semantic information embedded in speech, which is crucial for improving intelligibility, speaker consistency, and overall quality of enhanced speech signals. To enrich the SE model with semantic information, we employ language models as an efficient semantic learner and propose a comprehensive framework tailored for language model-based speech enhancement, called \textit{GenSE}. Specifically, we approach SE as a conditional language modeling task rather than a continuous signal regression problem defined in existing works. This is achieved by tokenizing speech signals into semantic tokens using a pre-trained self-supervised model and into acoustic tokens using a custom-designed single-quantizer neural codec model. To improve the stability of language model predictions, we propose a hierarchical modeling method that decouples the generation of clean semantic tokens and clean acoustic tokens into two distinct stages. Moreover, we introduce a token chain prompting mechanism during the acoustic token generation stage to ensure timbre consistency throughout the speech enhancement process. Experimental results on benchmark datasets demonstrate that our proposed approach outperforms state-of-the-art SE systems in terms of speech quality and generalization capability.