Measuring the Impact of Individual Domain Factors in Self-Supervised Pre-Training

TL;DR

This study investigates how individual domain factors like phonetics and syntax influence the effectiveness of self-supervised pre-training for speech recognition, revealing phonetic factors are most impactful.

Contribution

It provides a controlled analysis of individual domain factors in speech pre-training, highlighting the importance of phonetic aspects over syntactic or grammatical ones.

Findings

Phonetic factors significantly impact pre-training effectiveness.

Grammatical and syntactic factors are less influential.

First study to dissect domain factors in speech self-supervised pre-training.

Abstract

Human speech data comprises a rich set of domain factors such as accent, syntactic and semantic variety, or acoustic environment. Previous work explores the effect of domain mismatch in automatic speech recognition between pre-training and fine-tuning as a whole but does not dissect the contribution of individual factors. In this paper, we present a controlled study to better understand the effect of such factors on the performance of pre-trained representations on automatic speech recognition. To do so, we pre-train models either on modified natural speech or synthesized audio, with a single domain factor modified, and then measure performance after fine-tuning. Results show that phonetic domain factors play an important role during pre-training while grammatical and syntactic factors are far less important. To our knowledge, this is the first study to better understand the domain characteristics of pre-trained sets in self-supervised pre-training for speech.