Modeling Dependent Structure for Utterances in ASR Evaluation

TL;DR

This paper introduces a graphical lasso based method to explicitly model dependencies among utterances in ASR evaluation, improving the accuracy of significance testing for word error rate by identifying uncorrelated blocks.

Contribution

It presents a novel approach combining graphical lasso with blockwise bootstrap to rigorously identify uncorrelated utterance blocks for more reliable ASR evaluation.

Findings

Variance estimator of WER is statistically consistent.

Proposed method validated on LibriSpeech dataset.

Improves significance testing accuracy in dependent speech data.

Abstract

The bootstrap resampling method has been popular for performing significance analysis on word error rate (WER) in automatic speech recognition (ASR) evaluation. To deal with dependent speech data, the blockwise bootstrap approach is also introduced. By dividing utterances into uncorrelated blocks, this approach resamples these blocks instead of original data. However, it is typically nontrivial to uncover the dependent structure among utterances and identify the blocks, which might lead to subjective conclusions in statistical testing. In this paper, we present graphical lasso based methods to explicitly model such dependency and estimate uncorrelated blocks of utterances in a rigorous way, after which blockwise bootstrap is applied on top of the inferred blocks. We show the resulting variance estimator of WER in ASR evaluation is statistically consistent under mild conditions. We also demonstrate the validity of proposed approach on LibriSpeech dataset.