Data Selection Effects on Self-Supervised Learning of Audio Representations for French Audiovisual Broadcasts

TL;DR

This study investigates how the diversity of pretraining datasets affects self-supervised audio models' performance on various tasks, emphasizing the benefits of diverse broadcast content and data deduplication.

Contribution

It introduces a large, diverse broadcast audio corpus for SSL pretraining and evaluates its impact on multiple downstream audio tasks, highlighting the importance of dataset diversity and data deduplication.

Findings

Pretraining on diverse broadcast content improves downstream task performance.

Data deduplication reduces model memorization of training data.

Unified training can bridge speech and music audio understanding.

Abstract

Audio and speech self-supervised encoder models are now widely used for a lot of different tasks. Many of these models are often trained on clean segmented speech content such as LibriSpeech. In this paper, we look into how the pretraining datasets of such SSL (Self-Supervised Learning) models impact their downstream results. We build a large pretraining corpus of highly diverse TV and Radio broadcast audio content, which we describe with automatic tools. We use these annotations to build smaller subsets, which we use to train audio SSL models. Then, we evaluate the models on multiple downstream tasks such as automatic speech recognition, voice activity and music detection, or speaker recognition. The results show the potential of pretraining SSL models on diverse audio content without restricting it to speech. We also perform a membership inference attack to evaluate the encoder ability to memorize their training datasets, which highlight the importance of data deduplication. This unified training could bridge speech and music machine learning communities.