Should We Always Separate?: Switching Between Enhanced and Observed Signals for Overlapping Speech Recognition

TL;DR

This paper investigates whether speech separation always benefits automatic speech recognition in overlapping speech scenarios, finding that sometimes processing artifacts from enhancement can harm ASR performance, and proposes a switching method based on signal quality metrics.

Contribution

The paper provides an analysis showing when speech enhancement degrades ASR and introduces a simple switching algorithm to improve recognition accuracy.

Findings

Switching between observed and enhanced speech can improve ASR performance.

Speech enhancement may degrade ASR under certain noise and interference conditions.

A simple signal-based switching method outperforms always using separation or observed speech alone.

Abstract

Although recent advances in deep learning technology improved automatic speech recognition (ASR), it remains difficult to recognize speech when it overlaps other people's voices. Speech separation or extraction is often used as a front-end to ASR to handle such overlapping speech. However, deep neural network-based speech enhancement can generate `processing artifacts' as a side effect of the enhancement, which degrades ASR performance. For example, it is well known that single-channel noise reduction for non-speech noise (non-overlapping speech) often does not improve ASR. Likewise, the processing artifacts may also be detrimental to ASR in some conditions when processing overlapping speech with a separation/extraction method, although it is usually believed that separation/extraction improves ASR. In order to answer the question `Do we always have to separate/extract speech from mixtures?', we analyze ASR performance on observed and enhanced speech at various noise and interference conditions, and show that speech enhancement degrades ASR under some conditions even for overlapping speech. Based on these findings, we propose a simple switching algorithm between observed and enhanced speech based on the estimated signal-to-interference ratio and signal-to-noise ratio. We demonstrated experimentally that such a simple switching mechanism can improve recognition performance when processing artifacts are detrimental to ASR.