A Study of Data Selection Strategies for Pre-training Self-Supervised Speech Models

TL;DR

This paper investigates data selection strategies for pre-training self-supervised speech models, revealing that prioritizing longer utterances improves ASR performance and efficiency more than diversity or quantity.

Contribution

It demonstrates that data length is more crucial than diversity or size in pre-training SSL speech models, challenging common assumptions.

Findings

Prioritizing longer utterances improves ASR performance.

Using only half the dataset with longer utterances reduces pre-training time by 24%.

Data diversity does not significantly impact ASR results when selecting data.

Abstract

Self-supervised learning (SSL) has transformed speech processing, yet its reliance on massive pre-training datasets remains a bottleneck. While robustness is often attributed to scale and diversity, the role of the data distribution is less understood. We systematically examine how curated subsets of pre-training data influence Automatic Speech Recognition (ASR) performance. Surprisingly, optimizing for acoustic, speaker, or linguistic diversity yields no clear improvements over random sampling. Instead, we find that prioritizing the longest utterances achieves superior ASR results while using only half the original dataset, reducing pre-training time by 24% on a large corpora. These findings suggest that for pre-training speech SSL models, data length is a more critical factor than either data diversity or overall data quantity for performance and efficiency, offering a new perspective for data selection strategies in SSL speech processing.