End-to-End Integration of Speech Recognition, Dereverberation, Beamforming, and Self-Supervised Learning Representation

TL;DR

This paper introduces a novel end-to-end neural network architecture that integrates dereverberation, beamforming, SSLR, and ASR for multi-channel noisy speech recognition, achieving state-of-the-art results on CHiME-4 and REVERB datasets.

Contribution

It presents the first end-to-end system combining dereverberation, beamforming, SSLR, and ASR for multi-channel speech, improving recognition accuracy in noisy environments.

Findings

Achieved 1.77% WER on CHiME-4 6-channel track.

End-to-end integration outperforms separate modules.

Validated effectiveness on REVERB dataset.

Abstract

Self-supervised learning representation (SSLR) has demonstrated its significant effectiveness in automatic speech recognition (ASR), mainly with clean speech. Recent work pointed out the strength of integrating SSLR with single-channel speech enhancement for ASR in noisy environments. This paper further advances this integration by dealing with multi-channel input. We propose a novel end-to-end architecture by integrating dereverberation, beamforming, SSLR, and ASR within a single neural network. Our system achieves the best performance reported in the literature on the CHiME-4 6-channel track with a word error rate (WER) of 1.77%. While the WavLM-based strong SSLR demonstrates promising results by itself, the end-to-end integration with the weighted power minimization distortionless response beamformer, which simultaneously performs dereverberation and denoising, improves WER significantly. Its effectiveness is also validated on the REVERB dataset.