A Conformer-based ASR Frontend for Joint Acoustic Echo Cancellation, Speech Enhancement and Speech Separation

TL;DR

This paper introduces a Conformer-based frontend that jointly performs acoustic echo cancellation, speech enhancement, and speech separation, significantly improving ASR robustness in noisy and echo conditions by using a multi-input neural network.

Contribution

The novel joint model integrates three speech processing tasks into a single neural network, maintaining near-task-specific performance and enhancing robustness in challenging environments.

Findings

Reduces word error rate by at least 71% in low SNR conditions.

Performs within 10% of task-specific models on echo cancellation.

Significantly improves ASR accuracy in noisy and multi-speaker scenarios.

Abstract

We present a frontend for improving robustness of automatic speech recognition (ASR), that jointly implements three modules within a single model: acoustic echo cancellation, speech enhancement, and speech separation. This is achieved by using a contextual enhancement neural network that can optionally make use of different types of side inputs: (1) a reference signal of the playback audio, which is necessary for echo cancellation; (2) a noise context, which is useful for speech enhancement; and (3) an embedding vector representing the voice characteristic of the target speaker of interest, which is not only critical in speech separation, but also helpful for echo cancellation and speech enhancement. We present detailed evaluations to show that the joint model performs almost as well as the task-specific models, and significantly reduces word error rate in noisy conditions even when using a large-scale state-of-the-art ASR model. Compared to the noisy baseline, the joint model reduces the word error rate in low signal-to-noise ratio conditions by at least 71% on our echo cancellation dataset, 10% on our noisy dataset, and 26% on our multi-speaker dataset. Compared to task-specific models, the joint model performs within 10% on our echo cancellation dataset, 2% on the noisy dataset, and 3% on the multi-speaker dataset.