Speech-enhanced and Noise-aware Networks for Robust Speech Recognition

TL;DR

This paper introduces a noise-aware training framework with cascaded neural structures that jointly optimize speech enhancement and recognition, significantly reducing word error rates in noisy conditions.

Contribution

It proposes a novel noise-aware training framework combining a multi-task autoencoder with acoustic modeling, improving robustness of speech recognition systems.

Findings

Achieved 3.55% WER on Aurora-4 with CNN-TDNNF model.

Reduced WER by up to 33.53% compared to existing systems.

Outperformed baseline models on the AMI task.

Abstract

Compensation for channel mismatch and noise interference is essential for robust automatic speech recognition. Enhanced speech has been introduced into the multi-condition training of acoustic models to improve their generalization ability. In this paper, a noise-aware training framework based on two cascaded neural structures is proposed to jointly optimize speech enhancement and speech recognition. The feature enhancement module is composed of a multi-task autoencoder, where noisy speech is decomposed into clean speech and noise. By concatenating its enhanced, noise-aware, and noisy features for each frame, the acoustic-modeling module maps each feature-augmented frame into a triphone state by optimizing the lattice-free maximum mutual information and cross entropy between the predicted and actual state sequences. On top of the factorized time delay neural network (TDNN-F) and its convolutional variant (CNN-TDNNF), both with SpecAug, the two proposed systems achieve word error rate (WER) of 3.90% and 3.55%, respectively, on the Aurora-4 task. Compared with the best existing systems that use bigram and trigram language models for decoding, the proposed CNN-TDNNF-based system achieves a relative WER reduction of 15.20% and 33.53%, respectively. In addition, the proposed CNN-TDNNF-based system also outperforms the baseline CNN-TDNNF system on the AMI task.