Analyzing ASR pretraining for low-resource speech-to-text translation

TL;DR

This paper investigates how different factors like language relatedness, data size, and data augmentation influence the effectiveness of ASR pretraining for low-resource speech-to-text translation, revealing that WER and phonetic encoding are key predictors.

Contribution

It systematically analyzes the impact of pretraining data size, language relatedness, and data augmentation on low-resource AST, highlighting the importance of WER and phonetic encoding as predictors.

Findings

Best predictor of AST performance is the WER of the pretrained ASR model.

Differences in performance correlate with phonetic encoding in RNN layers.

Pretraining and data augmentation provide complementary improvements.

Abstract

Previous work has shown that for low-resource source languages, automatic speech-to-text translation (AST) can be improved by pretraining an end-to-end model on automatic speech recognition (ASR) data from a high-resource language. However, it is not clear what factors --e.g., language relatedness or size of the pretraining data-- yield the biggest improvements, or whether pretraining can be effectively combined with other methods such as data augmentation. Here, we experiment with pretraining on datasets of varying sizes, including languages related and unrelated to the AST source language. We find that the best predictor of final AST performance is the word error rate of the pretrained ASR model, and that differences in ASR/AST performance correlate with how phonetic information is encoded in the later RNN layers of our model. We also show that pretraining and data augmentation yield complementary benefits for AST.