Pre-training End-to-end ASR Models with Augmented Speech Samples Queried by Text

TL;DR

This paper introduces a low-cost method for augmenting speech data using unpaired speech segments and text, significantly improving end-to-end ASR performance without additional speech data.

Contribution

The paper presents a novel approach to generate augmented speech samples from unpaired data for pre-training end-to-end ASR models, reducing data requirements.

Findings

8.7% relative WER reduction with augmented data

Achieves similar performance as models trained on 75,000 hours of multilingual data

12.2% relative WER reduction when combining augmented and multilingual data

Abstract

In end-to-end automatic speech recognition system, one of the difficulties for language expansion is the limited paired speech and text training data. In this paper, we propose a novel method to generate augmented samples with unpaired speech feature segments and text data for model pre-training, which has the advantage of low cost without using additional speech data. When mixing 20,000 hours augmented speech data generated by our method with 12,500 hours original transcribed speech data for Italian Transformer transducer model pre-training, we achieve 8.7% relative word error rate reduction. The pre-trained model achieves similar performance as the model pre-trained with multilingual transcribed 75,000 hours raw speech data. When merging the augmented speech data with the multilingual data to pre-train a new model, we achieve even more relative word error rate reduction of 12.2% over the baseline, which further verifies the effectiveness of our method for speech data augmentation.