Improving CTC-based speech recognition via knowledge transferring from pre-trained language models

TL;DR

This paper introduces two novel methods for transferring knowledge from pre-trained language models like BERT and GPT2 to enhance CTC-based speech recognition models, significantly reducing error rates without external LMs.

Contribution

The paper proposes two innovative knowledge transfer techniques from pre-trained LMs to improve CTC-based speech recognition, addressing their inherent weaknesses.

Findings

Achieved a CER of 4.2% on AISHELL-1 test set.

Reduced CER by 16.1% relative compared to vanilla CTC models.

Improved speech recognition performance without external language models.

Abstract

Recently, end-to-end automatic speech recognition models based on connectionist temporal classification (CTC) have achieved impressive results, especially when fine-tuned from wav2vec2.0 models. Due to the conditional independence assumption, CTC-based models are always weaker than attention-based encoder-decoder models and require the assistance of external language models (LMs). To solve this issue, we propose two knowledge transferring methods that leverage pre-trained LMs, such as BERT and GPT2, to improve CTC-based models. The first method is based on representation learning, in which the CTC-based models use the representation produced by BERT as an auxiliary learning target. The second method is based on joint classification learning, which combines GPT2 for text modeling with a hybrid CTC/attention architecture. Experiment on AISHELL-1 corpus yields a character error rate (CER) of 4.2% on the test set. When compared to the vanilla CTC-based models fine-tuned from the wav2vec2.0 models, our knowledge transferring method reduces CER by 16.1% relatively without external LMs.