Generalizing RNN-Transducer to Out-Domain Audio via Sparse Self-Attention Layers

TL;DR

This paper improves out-domain speech recognition by introducing sparse self-attention layers in Conformer RNN-T models, significantly reducing errors on long-form utterances through targeted modifications and a state reset method.

Contribution

It proposes sparse self-attention layers and a state reset technique to enhance Conformer RNN-T's out-domain generalization, addressing domain mismatch issues.

Findings

27.6% relative CER reduction on out-domain test data

Sparse self-attention effectively mitigates long-form utterance errors

Enhanced out-domain robustness over fully connected self-attention models

Abstract

Recurrent neural network transducer (RNN-T) is an end-to-end speech recognition framework converting input acoustic frames into a character sequence. The state-of-the-art encoder network for RNN-T is the Conformer, which can effectively model the local-global context information via its convolution and self-attention layers. Although Conformer RNN-T has shown outstanding performance, most studies have been verified in the setting where the train and test data are drawn from the same domain. The domain mismatch problem for Conformer RNN-T has not been intensively investigated yet, which is an important issue for the product-level speech recognition system. In this study, we identified that fully connected self-attention layers in the Conformer caused high deletion errors, specifically in the long-form out-domain utterances. To address this problem, we introduce sparse self-attention layers for Conformer-based encoder networks, which can exploit local and generalized global information by pruning most of the in-domain fitted global connections. Also, we propose a state reset method for the generalization of the prediction network to cope with long-form utterances. Applying proposed methods to an out-domain test, we obtained 27.6% relative character error rate (CER) reduction compared to the fully connected self-attention layer-based Conformers.