Distortionless Multi-Channel Target Speech Enhancement for Overlapped Speech Recognition

TL;DR

This paper introduces a multi-channel convolutional network for speech enhancement that minimizes distortion, thereby improving recognition accuracy in overlapped, noisy, and far-field speech scenarios.

Contribution

It proposes three novel approaches for distortionless speech enhancement, including complex ratio mask estimation, multi-objective learning with fbank loss, and joint finetuning with an acoustic model.

Findings

All approaches reduce speech distortion effectively.

Enhanced models improve speech recognition accuracy.

Joint tuning with acoustic models yields best results.

Abstract

Speech enhancement techniques based on deep learning have brought significant improvement on speech quality and intelligibility. Nevertheless, a large gain in speech quality measured by objective metrics, such as perceptual evaluation of speech quality (PESQ), does not necessarily lead to improved speech recognition performance due to speech distortion in the enhancement stage. In this paper, a multi-channel dilated convolutional network based frequency domain modeling is presented to enhance target speaker in the far-field, noisy and multi-talker conditions. We study three approaches towards distortionless waveforms for overlapped speech recognition: estimating complex ideal ratio mask with an infinite range, incorporating the fbank loss in a multi-objective learning and finetuning the enhancement model by an acoustic model. Experimental results proved the effectiveness of all three approaches on reducing speech distortions and improving recognition accuracy. Particularly, the jointly tuned enhancement model works very well with other standalone acoustic model on real test data.