Self-supervised learning of speech representations with Dutch archival data

TL;DR

This study leverages Dutch archival TV broadcast data to improve self-supervised speech models, addressing data quality, pre-processing, and multilingual training, resulting in a state-of-the-art Dutch wav2vec 2.0 model.

Contribution

It introduces effective pre-processing strategies and demonstrates that monolingual pre-training yields more robust speech representations for Dutch.

Findings

Music, noise, and speaker overlap impact SSL convergence.

Pre-processing with Whisper improves data quality for SSL.

Monolingual pre-training outperforms multilingual in robustness.

Abstract

This paper explores the use of Dutch archival television broadcast data for self-supervised learning of speech foundation models, specifically wav2vec 2.0. We first study data quality assumptions for pre-training, and show how music, noise and speaker overlap affect SSL convergence and downstream fine-tuning performance. Secondly, we explore effectively pre-processing strategies to convert the noisy broadcast dataset into a qualitative dataset for pre-training, by using Whisper and WhisperX. Thirdly, we compare mono-lingual and multi-lingual pre-training with equivalent amounts of data, and show that mono-lingual pre-training is more robust to out-of-domain data. Lastly, we achieve a state-of-the-art LARGE wav2vec 2.0 model for the Dutch language, by a continuation of pre-training a wav2vec 2.0 XLS-R model checkpoint with our 55k hour archival dataset.