Attentional Speech Recognition Models Misbehave on Out-of-domain Utterances

TL;DR

This paper investigates how attentional speech recognition models generate excessively long, repetitive outputs on out-of-domain utterances, revealing issues intrinsic to the attention mechanism and proposing a length prediction model to mitigate this problem.

Contribution

It identifies the problem of excessively long outputs in attentional ASR models on out-of-domain data and introduces a length prediction model to improve decoding robustness.

Findings

Attentional models produce overly long, repetitive transcripts on out-of-domain utterances.

Hybrid DNN-HMM models do not exhibit this problem, indicating a specific issue with attention mechanisms.

A length prediction model effectively identifies and truncates problematic outputs, maintaining accuracy.

Abstract

We discuss the problem of echographic transcription in autoregressive sequence-to-sequence attentional architectures for automatic speech recognition, where a model produces very long sequences of repetitive outputs when presented with out-of-domain utterances. We decode audio from the British National Corpus with an attentional encoder-decoder model trained solely on the LibriSpeech corpus. We observe that there are many 5-second recordings that produce more than 500 characters of decoding output (i.e. more than 100 characters per second). A frame-synchronous hybrid (DNN-HMM) model trained on the same data does not produce these unusually long transcripts. These decoding issues are reproducible in a speech transformer model from ESPnet, and to a lesser extent in a self-attention CTC model, suggesting that these issues are intrinsic to the use of the attention mechanism. We create a separate length prediction model to predict the correct number of wordpieces in the output, which allows us to identify and truncate problematic decoding results without increasing word error rates on the LibriSpeech task.