Phoneme-based Distribution Regularization for Speech Enhancement

TL;DR

This paper introduces a phoneme-based distribution regularization method that leverages phoneme information to improve speech enhancement quality and ASR accuracy by modulating feature distributions.

Contribution

It proposes a novel PbDr module that incorporates phoneme classification vectors into speech enhancement networks for better feature regularization.

Findings

Boosts perceptual speech quality

Improves ASR recognition accuracy

Can be integrated into existing networks

Abstract

Existing speech enhancement methods mainly separate speech from noises at the signal level or in the time-frequency domain. They seldom pay attention to the semantic information of a corrupted signal. In this paper, we aim to bridge this gap by extracting phoneme identities to help speech enhancement. Specifically, we propose a phoneme-based distribution regularization (PbDr) for speech enhancement, which incorporates frame-wise phoneme information into speech enhancement network in a conditional manner. As different phonemes always lead to different feature distributions in frequency, we propose to learn a parameter pair, i.e. scale and bias, through a phoneme classification vector to modulate the speech enhancement network. The modulation parameter pair includes not only frame-wise but also frequency-wise conditions, which effectively map features to phoneme-related distributions. In this way, we explicitly regularize speech enhancement features by recognition vectors. Experiments on public datasets demonstrate that the proposed PbDr module can not only boost the perceptual quality for speech enhancement but also the recognition accuracy of an ASR system on the enhanced speech. This PbDr module could be readily incorporated into other speech enhancement networks as well.