From Continuous to Discrete: Cross-Domain Collaborative General Speech Enhancement via Hierarchical Language Models

TL;DR

OmniGSE is a two-stage framework that combines continuous feature enhancement with discrete token reconstruction using hierarchical language models, enabling effective handling of complex, multi-distortion speech scenarios.

Contribution

The paper introduces OmniGSE, a novel cross-domain speech enhancement framework that integrates discriminative and generative methods through hierarchical language models for robust real-world performance.

Findings

Outperforms existing models on multiple benchmarks

Excels in scenarios with compound distortions

Demonstrates robustness across diverse real-world distortions

Abstract

This paper introduces OmniGSE, a novel general speech enhancement (GSE) framework designed to mitigate the diverse distortions that speech signals encounter in real-world scenarios. These distortions include background noise, reverberation, bandwidth limitations, signal clipping, and network packet loss. Existing methods typically focus on optimizing for a single type of distortion, often struggling to effectively handle the simultaneous presence of multiple distortions in complex scenarios. OmniGSE bridges this gap by integrating the strengths of discriminative and generative approaches through a two-stage architecture that enables cross-domain collaborative optimization. In the first stage, continuous features are enhanced using a lightweight channel-split NAC-RoFormer. In the second stage, discrete tokens are generated to reconstruct high-quality speech through language models. Specifically, we designed a hierarchical language model structure consisting of a RootLM and multiple BranchLMs. The RootLM models general acoustic features across codebook layers, while the BranchLMs explicitly capture the progressive relationships between different codebook levels. Experimental results demonstrate that OmniGSE surpasses existing models across multiple benchmarks, particularly excelling in scenarios involving compound distortions. These findings underscore the framework's potential for robust and versatile speech enhancement in real-world applications.