HuBERT-VIC: Improving Noise-Robust Automatic Speech Recognition of Speech Foundation Model via Variance-Invariance-Covariance Regularization

TL;DR

HuBERT-VIC introduces variance-invariance-covariance regularization to enhance noise robustness in speech foundation models, significantly improving performance on noisy speech recognition tasks.

Contribution

This paper presents HuBERT-VIC, a novel regularization method that improves noise robustness in speech foundation models by adjusting statistical properties of speech representations.

Findings

23.3% relative improvement on LibriSpeech test-clean

13.2% relative improvement on test-other

Enhanced generalization across different noise types

Abstract

Noise robustness in speech foundation models (SFMs) has been a critical challenge, as most models are primarily trained on clean data and experience performance degradation when the models are exposed to noisy speech. To address this issue, we propose HuBERT-VIC, a noise-robust SFM with variance, in-variance, and covariance regularization (VICReg) objectives. These objectives adjust the statistics of noisy speech representations, enabling the model to capture diverse acoustic characteristics and improving the generalization ability across different types of noise. When applied to HuBERT, our model shows relative performance improvements of 23.3% on LibriSpeech test-clean and 13.2% on test-other, compared to the baseline model pre-trained on noisy speech.