ASR Rescoring and Confidence Estimation with ELECTRA

TL;DR

This paper introduces a novel ASR error detection and rescoring method using ELECTRA and an extended version called P-ELECTRA, which leverages phonetic information for improved accuracy and confidence estimation.

Contribution

It proposes a new error detection approach with ELECTRA and P-ELECTRA, tailored for ASR, enhancing rescoring and confidence estimation over traditional language models.

Findings

ELECTRA-based rescoring is competitive with traditional methods and faster.

P-ELECTRA improves error detection by incorporating phonetic information.

ELECTRA outperforms BERT in confidence estimation tasks.

Abstract

In automatic speech recognition (ASR) rescoring, the hypothesis with the fewest errors should be selected from the n-best list using a language model (LM). However, LMs are usually trained to maximize the likelihood of correct word sequences, not to detect ASR errors. We propose an ASR rescoring method for directly detecting errors with ELECTRA, which is originally a pre-training method for NLP tasks. ELECTRA is pre-trained to predict whether each word is replaced by BERT or not, which can simulate ASR error detection on large text corpora. To make this pre-training closer to ASR error detection, we further propose an extended version of ELECTRA called phone-attentive ELECTRA (P-ELECTRA). In the pre-training of P-ELECTRA, each word is replaced by a phone-to-word conversion model, which leverages phone information to generate acoustically similar words. Since our rescoring method is optimized for detecting errors, it can also be used for word-level confidence estimation. Experimental evaluations on the Librispeech and TED-LIUM2 corpora show that our rescoring method with ELECTRA is competitive with conventional rescoring methods with faster inference. ELECTRA also performs better in confidence estimation than BERT because it can learn to detect inappropriate words not only in fine-tuning but also in pre-training.