Spectral feature mapping with mimic loss for robust speech recognition

TL;DR

This paper introduces a global mimic loss criterion for speech enhancement that aligns de-noised speech with phonetic structures, improving downstream speech recognition performance.

Contribution

It proposes a novel mimic loss that guides speech enhancement models to produce phonemically meaningful outputs, enhancing ASR accuracy.

Findings

Significant WER reduction on CHiME-2 corpus

Effective integration of spectral classifier with speech enhancer

Improved phonetic consistency in de-noised speech

Abstract

For the task of speech enhancement, local learning objectives are agnostic to phonetic structures helpful for speech recognition. We propose to add a global criterion to ensure de-noised speech is useful for downstream tasks like ASR. We first train a spectral classifier on clean speech to predict senone labels. Then, the spectral classifier is joined with our speech enhancer as a noisy speech recognizer. This model is taught to imitate the output of the spectral classifier alone on clean speech. This \textit{mimic loss} is combined with the traditional local criterion to train the speech enhancer to produce de-noised speech. Feeding the de-noised speech to an off-the-shelf Kaldi training recipe for the CHiME-2 corpus shows significant improvements in WER.