Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data

TL;DR

This paper introduces Speech2C, a novel pre-training method for encoder-decoder ASR models using unpaired speech data, which improves recognition accuracy by learning to reconstruct pseudo codes before generating text.

Contribution

It proposes a new pre-training approach with two tasks using pseudo codes, enhancing encoder-decoder ASR performance with unpaired speech data.

Findings

Reduces WER by 19.2% compared to no decoder pre-training

Outperforms wav2vec 2.0 and HuBERT on small datasets

Effective in low-resource speech recognition scenarios

Abstract

This paper studies a novel pre-training technique with unpaired speech data, Speech2C, for encoder-decoder based automatic speech recognition (ASR). Within a multi-task learning framework, we introduce two pre-training tasks for the encoder-decoder network using acoustic units, i.e., pseudo codes, derived from an offline clustering model. One is to predict the pseudo codes via masked language modeling in encoder output, like HuBERT model, while the other lets the decoder learn to reconstruct pseudo codes autoregressively instead of generating textual scripts. In this way, the decoder learns to reconstruct original speech information with codes before learning to generate correct text. Comprehensive experiments on the LibriSpeech corpus show that the proposed Speech2C can relatively reduce the word error rate (WER) by 19.2% over the method without decoder pre-training, and also outperforms significantly the state-of-the-art wav2vec 2.0 and HuBERT on fine-tuning subsets of 10h and 100h. We release our code and model at https://github.com/microsoft/SpeechT5/tree/main/Speech2C.