Data-Centric Lessons To Improve Speech-Language Pretraining

TL;DR

This paper investigates how data processing and curation strategies impact speech-language model pretraining, demonstrating that careful data management can significantly enhance performance even with smaller models.

Contribution

It provides a systematic data-centric analysis for SpeechLM pretraining, introducing effective data processing, augmentation, and sequencing techniques that improve model performance.

Findings

Data curation significantly boosts SpeechLM performance.

Synthetic data augmentation enhances model capabilities.

Proper data sequencing improves training efficiency.

Abstract

Spoken Question-Answering (SQA) is a core capability for useful and interactive artificial intelligence systems. Recently, several speech-language models (SpeechLMs) have been released with a specific focus on improving their SQA performance. However, a lack of controlled ablations of pretraining data processing and curation makes it challenging to understand what factors account for performance, despite substantial gains from similar studies in other data modalities. In this work, we address this gap by conducting a data-centric exploration for pretraining SpeechLMs. We focus on three research questions fundamental to speech-language pretraining data: (1) how to process raw web-crawled audio content for speech-text pretraining, (2) how to construct synthetic pretraining datasets to augment web-crawled data and (3) how to interleave (text, audio) segments into training sequences. We apply the insights from our controlled data-centric ablations to pretrain a 3.8B-parameter SpeechLM, called SpeLangy, that outperforms models that are up to 3x larger by 10.2% absolute performance. We hope our findings highlight the impact of effective data curation for speech-language pretraining and guide future data-centric exploration in SpeechLMs.