Analyzing the impact of speaker localization errors on speech separation for automatic speech recognition

TL;DR

This paper examines how errors in speaker localization affect speech separation and recognition accuracy in multispeaker environments, highlighting the importance of accurate localization for reducing word error rates.

Contribution

It introduces a pipeline combining delay-and-sum beamforming, neural network-based masking, and adaptive beamforming to analyze localization errors' impact on speech recognition.

Findings

Ground truth localization yields 29.4% WER

Estimated localization yields 42.4% WER

Higher SIR significantly reduces WER

Abstract

We investigate the effect of speaker localization on the performance of speech recognition systems in a multispeaker, multichannel environment. Given the speaker location information, speech separation is performed in three stages. In the first stage, a simple delay-and-sum (DS) beamformer is used to enhance the signal impinging from the speaker location which is then used to estimate a time-frequency mask corresponding to the localized speaker using a neural network. This mask is used to compute the second order statistics and to derive an adaptive beamformer in the third stage. We generated a multichannel, multispeaker, reverberated, noisy dataset inspired from the well studied WSJ0-2mix and study the performance of the proposed pipeline in terms of the word error rate (WER). An average WER of $29.4$% was achieved using the ground truth localization information and $42.4$% using the localization information estimated via GCC-PHAT. The signal-to-interference ratio (SIR) between the speakers has a higher impact on the ASR performance, to the extent of reducing the WER by $59$% relative for a SIR increase of $15$ dB. By contrast, increasing the spatial distance to $50^\circ$ or more improves the WER by $23$% relative only