Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition

TL;DR

This paper explores adapting pre-trained language models like GPT, GPT-2, and BERT for speech recognition, proposing a conversion method for BERT's bidirectional outputs and demonstrating significant WERR improvements on ASR tasks.

Contribution

It introduces a novel conversion method for BERT's bidirectional outputs and combines multiple pre-trained models to enhance speech recognition accuracy.

Findings

GPT and GPT-2 combination outperforms models trained from scratch by up to 12% WERR.

Conversion method enables BERT to improve WERR by 3%.

Combining BERT, GPT, and GPT-2 yields further WERR improvements.

Abstract

Language models (LMs) pre-trained on massive amounts of text, in particular bidirectional encoder representations from Transformers (BERT), generative pre-training (GPT), and GPT-2, have become a key technology for many natural language processing tasks. In this paper, we present results using fine-tuned GPT, GPT-2, and their combination for automatic speech recognition (ASR). Unlike unidirectional LM GPT and GPT-2, BERT is bidirectional whose direct product of the output probabilities is no longer a valid language prior probability. A conversion method is proposed to compute the correct language prior probability based on bidirectional LM outputs in a mathematically exact way. Experimental results on the widely used AMI and Switchboard ASR tasks showed that the combination of the fine-tuned GPT and GPT-2 outperformed the combination of three neural LMs with different architectures trained from scratch on the in-domain text by up to a 12% relative word error rate reduction (WERR). Furthermore, on the AMI corpus, the proposed conversion for language prior probabilities enables BERT to obtain an extra 3% relative WERR, and the combination of BERT, GPT and GPT-2 results in further improvements.