A Semantic Information-based Hierarchical Speech Enhancement Method Using Factorized Codec and Diffusion Model

TL;DR

This paper introduces a hierarchical speech enhancement method that leverages semantic information and a factorized codec with diffusion models to improve speech quality and downstream TTS performance in noisy environments.

Contribution

It presents a novel hierarchical approach that separately models semantic and acoustic attributes using a factorized codec and diffusion model, enhancing robustness in complex acoustic scenarios.

Findings

Outperforms state-of-the-art baselines in speech quality

Improves downstream TTS performance in noisy conditions

Demonstrates robustness in challenging acoustic environments

Abstract

Most current speech enhancement (SE) methods recover clean speech from noisy inputs by directly estimating time-frequency masks or spectrums. However, these approaches often neglect the distinct attributes, such as semantic content and acoustic details, inherent in speech signals, which can hinder performance in downstream tasks. Moreover, their effectiveness tends to degrade in complex acoustic environments. To overcome these challenges, we propose a novel, semantic information-based, step-by-step factorized SE method using factorized codec and diffusion model. Unlike traditional SE methods, our hierarchical modeling of semantic and acoustic attributes enables more robust clean speech recovery, particularly in challenging acoustic scenarios. Moreover, this method offers further advantages for downstream TTS tasks. Experimental results demonstrate that our algorithm not only outperforms SOTA baselines in terms of speech quality but also enhances TTS performance in noisy environments.