PASE: Leveraging the Phonological Prior of WavLM for Low-Hallucination Generative Speech Enhancement

TL;DR

This paper introduces PASE, a speech enhancement framework that leverages the phonological prior from WavLM to reduce hallucinations and improve perceptual quality in noisy speech.

Contribution

The paper proposes a novel generative speech enhancement method that uses WavLM's phonological prior and dual-stream vocoder training to mitigate hallucinations and enhance speech quality.

Findings

PASE outperforms state-of-the-art models in perceptual quality.

Significantly reduces linguistic hallucinations compared to prior methods.

Achieves lower acoustic hallucinations while maintaining speech naturalness.

Abstract

Generative models have shown remarkable performance in speech enhancement (SE), achieving superior perceptual quality over traditional discriminative approaches. However, existing generative SE approaches often overlook the risk of hallucination under severe noise, leading to incorrect spoken content or inconsistent speaker characteristics, which we term linguistic and acoustic hallucinations, respectively. We argue that linguistic hallucination stems from models' failure to constrain valid phonological structures and it is a more fundamental challenge. While language models (LMs) are well-suited for capturing the underlying speech structure through modeling the distribution of discrete tokens, existing approaches are limited in learning from noise-corrupted representations, which can lead to contaminated priors and hallucinations. To overcome these limitations, we propose the Phonologically Anchored Speech Enhancer (PASE), a generative SE framework that leverages the robust phonological prior embedded in the pre-trained WavLM model to mitigate hallucinations. First, we adapt WavLM into a denoising expert via representation distillation to clean its final-layer features. Guided by the model's intrinsic phonological prior, this process enables robust denoising while minimizing linguistic hallucinations. To further reduce acoustic hallucinations, we train the vocoder with a dual-stream representation: the high-level phonetic representation provides clean linguistic content, while a low-level acoustic representation retains speaker identity and prosody. Experimental results demonstrate that PASE not only surpasses state-of-the-art discriminative models in perceptual quality, but also significantly outperforms prior generative models with substantially lower linguistic and acoustic hallucinations.