Aligning Generative Speech Enhancement with Perceptual Feedback

TL;DR

This paper introduces a novel speech enhancement method that aligns model training with human perceptual preferences using neural MOS predictors and Direct Preference Optimization, leading to significant quality improvements.

Contribution

It is the first to incorporate perceptual feedback into LM-based speech enhancement and applies DPO with UTMOS to directly optimize for perceptual quality.

Findings

Achieved up to 56% relative improvement in speech quality metrics.

First integration of perceptual feedback in LM-based speech enhancement.

Demonstrated broad applicability of the approach within LM frameworks.

Abstract

Language Model (LM)-based speech enhancement (SE) has recently emerged as a promising direction, but existing approaches predominantly rely on token-level likelihood objectives that weakly reflect human perception. This mismatch limits progress, as optimizing signal accuracy does not always improve naturalness or listening comfort. We address this gap by introducing a perceptually aligned LM-based SE approach. Our method applies Direct Preference Optimization (DPO) with UTMOS, a neural MOS predictor, as a proxy for human ratings, directly steering models toward perceptually preferred outputs. This design directly connects model training to perceptual quality and is broadly applicable within LM-based SE frameworks. On the Deep Noise Suppression Challenge 2020 test sets, our approach consistently improves speech quality metrics, achieving relative gains of up to 56%. To our knowledge, this is the first integration of perceptual feedback into LM-based SE and the first application of DPO in the SE domain, establishing a new paradigm for perceptually aligned enhancement with SE.