RescoreBERT: Discriminative Speech Recognition Rescoring with BERT

TL;DR

RescoreBERT introduces a novel discriminative training approach for BERT-based speech recognition rescoring, significantly reducing word error rates and latency in ASR systems by integrating a minimum WER loss with pretrained language models.

Contribution

The paper presents a new method for training BERT with a discriminative MWER loss for ASR rescoring, combining MLM pretraining with discriminative fine-tuning.

Findings

Reduces WER by 6.6%/3.4% on LibriSpeech test sets.

Decreases latency and WER by 3-8% on conversational datasets.

Outperforms baseline BERT rescoring without discriminative training.

Abstract

Second-pass rescoring is an important component in automatic speech recognition (ASR) systems that is used to improve the outputs from a first-pass decoder by implementing a lattice rescoring or $n$-best re-ranking. While pretraining with a masked language model (MLM) objective has received great success in various natural language understanding (NLU) tasks, it has not gained traction as a rescoring model for ASR. Specifically, training a bidirectional model like BERT on a discriminative objective such as minimum WER (MWER) has not been explored. Here we show how to train a BERT-based rescoring model with MWER loss, to incorporate the improvements of a discriminative loss into fine-tuning of deep bidirectional pretrained models for ASR. Specifically, we propose a fusion strategy that incorporates the MLM into the discriminative training process to effectively distill knowledge from a pretrained model. We further propose an alternative discriminative loss. This approach, which we call RescoreBERT, reduces WER by 6.6%/3.4% relative on the LibriSpeech clean/other test sets over a BERT baseline without discriminative objective. We also evaluate our method on an internal dataset from a conversational agent and find that it reduces both latency and WER (by 3 to 8% relative) over an LSTM rescoring model.