Sequence-level Confidence Classifier for ASR Utterance Accuracy and Application to Acoustic Models

TL;DR

This paper introduces a sequence-level confidence classifier for ASR that correlates well with accuracy and remains stable across model updates, enabling improved data selection for acoustic model adaptation and semi-supervised learning.

Contribution

A novel sequence-level confidence classifier that is interpretable, stable, and highly correlated with ASR accuracy, facilitating better data selection for model adaptation.

Findings

Sequence-level confidence scores achieve high correlation with ASR accuracy.

Using confidence scores for data selection improves word error rate reduction.

Method is effective in both supervised and semi-supervised adaptation scenarios.

Abstract

Scores from traditional confidence classifiers (CCs) in automatic speech recognition (ASR) systems lack universal interpretation and vary with updates to the underlying confidence or acoustic models (AMs). In this work, we build interpretable confidence scores with an objective to closely align with ASR accuracy. We propose a new sequence-level CC with a richer context providing CC scores highly correlated with ASR accuracy and scores stable across CC updates. Hence, expanding CC applications. Recently, AM customization has gained traction with the widespread use of unified models. Conventional adaptation strategies that customize AM expect well-matched data for the target domain with gold-standard transcriptions. We propose a cost-effective method of using CC scores to select an optimal adaptation data set, where we maximize ASR gains from minimal data. We study data in various confidence ranges and optimally choose data for AM adaptation with KL-Divergence regularization. On the Microsoft voice search task, data selection for supervised adaptation using the sequence-level confidence scores achieves word error rate reduction (WERR) of 8.5% for row-convolution LSTM (RC-LSTM) and 5.2% for latency-controlled bidirectional LSTM (LC-BLSTM). In the semi-supervised case, with ASR hypotheses as labels, our method provides WERR of 5.9% and 2.8% for RC-LSTM and LC-BLSTM, respectively.