Training Autoregressive Speech Recognition Models with Limited in-domain Supervision

TL;DR

This paper investigates training autoregressive speech recognition models with limited in-domain supervision, leveraging self-supervised learning and open source data to outperform larger models in conversational speech tasks.

Contribution

It introduces a method combining self-supervised learning and semi-supervised training with open source data, showing improved performance over fine-tuned XLS-R models in limited data scenarios.

Findings

Smaller autoregressive models outperform fine-tuned XLS-R with limited in-domain data.

Using pseudotranscriptions reduces WER by up to 8% absolute.

Open source read speech data enhances model adaptation in conversational speech.

Abstract

Advances in self-supervised learning have significantly reduced the amount of transcribed audio required for training. However, the majority of work in this area is focused on read speech. We explore limited supervision in the domain of conversational speech. While we assume the amount of in-domain data is limited, we augment the model with open source read speech data. The XLS-R model has been shown to perform well with limited adaptation data and serves as a strong baseline. We use untranscribed data for self-supervised learning and semi-supervised training in an autoregressive encoder-decoder model. We demonstrate that by using the XLS-R model for pseudotranscription, a much smaller autoregressive model can outperform a finetuned XLS-R model when transcribed in-domain data is limited, reducing WER by as much as 8% absolute.