Word-level confidence estimation for RNN transducers

TL;DR

This paper introduces a lightweight neural confidence model for RNN transducer-based speech recognition that leverages time information and a novel mapping technique, achieving robust performance across various ASR configurations.

Contribution

The paper presents a novel, efficient confidence estimation method for RNN-T ASR systems utilizing time data and a simple mapping trick, improving calibration and robustness.

Findings

Achieves 0.4 NCE and 0.05 ECE on long-form test sets.

Robust across different ASR configurations and target types.

Highlights the importance of evaluation metrics for practical applications.

Abstract

Confidence estimate is an often requested feature in applications such as medical transcription where errors can impact patient care and the confidence estimate could be used to alert medical professionals to verify potential errors in recognition. In this paper, we present a lightweight neural confidence model tailored for Automatic Speech Recognition (ASR) system with Recurrent Neural Network Transducers (RNN-T). Compared to other existing approaches, our model utilizes: (a) the time information associated with recognized words, which reduces the computational complexity, and (b) a simple and elegant trick for mapping between sub-word and word sequences. The mapping addresses the non-unique tokenization and token deletion problems while amplifying differences between confusable words. Through extensive empirical evaluations on two different long-form test sets, we demonstrate that the model achieves a performance of 0.4 Normalized Cross Entropy (NCE) and 0.05 Expected Calibration Error (ECE). It is robust across different ASR configurations, including target types (graphemes vs. morphemes), traffic conditions (streaming vs. non-streaming), and encoder types. We further discuss the importance of evaluation metrics to reflect practical applications and highlight the need for further work in improving Area Under the Curve (AUC) for Negative Precision Rate (NPV) and True Negative Rate (TNR).