Combining Unsupervised and Text Augmented Semi-Supervised Learning for Low Resourced Autoregressive Speech Recognition

TL;DR

This paper explores combining unsupervised pretraining and text-augmented semi-supervised learning to improve low-resource autoregressive speech recognition, achieving significant reductions in word error rate.

Contribution

It introduces a novel approach that integrates unsupervised pretraining with semi-supervised learning and external language models for low-resource speech recognition.

Findings

Unsupervised pretraining outperforms traditional semi-supervised training.

Combining unsupervised and semi-supervised methods yields a 5% absolute WER improvement.

CTC-based decoding enhances the use of additional text data and improves performance.

Abstract

Recent advances in unsupervised representation learning have demonstrated the impact of pretraining on large amounts of read speech. We adapt these techniques for domain adaptation in low-resource -- both in terms of data and compute -- conversational and broadcast domains. Moving beyond CTC, we pretrain state-of-the-art Conformer models in an unsupervised manner. While the unsupervised approach outperforms traditional semi-supervised training, the techniques are complementary. Combining the techniques is a 5% absolute improvement in WER, averaged over all conditions, compared to semi-supervised training alone. Additional text data is incorporated through external language models. By using CTC-based decoding, we are better able to take advantage of the additional text data. When used as a transcription model, it allows the Conformer model to better incorporate the knowledge from the language model through semi-supervised training than shallow fusion. Final performance is an additional 2% better absolute when using CTC-based decoding for semi-supervised training compared to shallow fusion.