Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction

TL;DR

This paper introduces a noise-robust speech representation learning method combining contrastive learning with a reconstruction module, significantly improving ASR performance in noisy environments without requiring denoising during inference.

Contribution

It proposes a novel multi-task continual pre-training framework that enhances noise robustness of speech representations using a reconstruction module alongside contrastive learning.

Findings

Reduces WER by around 4.1/7.5% on noisy LibriSpeech test sets

Achieves state-of-the-art performance on CHiME-4 noisy speech recognition

Performs comparably to supervised methods with only 16% labeled data

Abstract

Noise robustness is essential for deploying automatic speech recognition (ASR) systems in real-world environments. One way to reduce the effect of noise interference is to employ a preprocessing module that conducts speech enhancement, and then feed the enhanced speech to an ASR backend. In this work, instead of suppressing background noise with a conventional cascaded pipeline, we employ a noise-robust representation learned by a refined self-supervised framework for noisy speech recognition. We propose to combine a reconstruction module with contrastive learning and perform multi-task continual pre-training on noisy data. The reconstruction module is used for auxiliary learning to improve the noise robustness of the learned representation and thus is not required during inference. Experiments demonstrate the effectiveness of our proposed method. Our model substantially reduces the word error rate (WER) for the synthesized noisy LibriSpeech test sets, and yields around 4.1/7.5% WER reduction on noisy clean/other test sets compared to data augmentation. For the real-world noisy speech from the CHiME-4 challenge (1-channel track), we have obtained the state of the art ASR performance without any denoising front-end. Moreover, we achieve comparable performance to the best supervised approach reported with only 16% of labeled data.