Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition

TL;DR

Wav2vec-Switch introduces a contrastive learning approach that enhances noise robustness in speech representations by training on original-noisy speech pairs, improving speech recognition accuracy in noisy conditions.

Contribution

It proposes a novel contrastive learning method that enforces consistent predictions between original and noisy speech, significantly improving noise robustness in speech recognition models.

Findings

Achieves 2.9-4.9% relative WER reduction on synthesized noisy data.

Attains 5.7% relative WER reduction on real noisy CHiME-4 data.

Outperforms baseline data augmentation and matches or surpasses speech enhancement methods.

Abstract

The goal of self-supervised learning (SSL) for automatic speech recognition (ASR) is to learn good speech representations from a large amount of unlabeled speech for the downstream ASR task. However, most SSL frameworks do not consider noise robustness which is crucial for real-world applications. In this paper we propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech via contrastive learning. Specifically, we feed original-noisy speech pairs simultaneously into the wav2vec 2.0 network. In addition to the existing contrastive learning task, we switch the quantized representations of the original and noisy speech as additional prediction targets of each other. By doing this, it enforces the network to have consistent predictions for the original and noisy speech, thus allows to learn contextualized representation with noise robustness. Our experiments on synthesized and real noisy data show the effectiveness of our method: it achieves 2.9--4.9% relative word error rate (WER) reduction on the synthesized noisy LibriSpeech data without deterioration on the original data, and 5.7% on CHiME-4 real 1-channel noisy data compared to a data augmentation baseline even with a strong language model for decoding. Our results on CHiME-4 can match or even surpass those with well-designed speech enhancement components.