ConvRNN-T: Convolutional Augmented Recurrent Neural Network Transducers for Streaming Speech Recognition

TL;DR

ConvRNN-T introduces a convolutional frontend to LSTM-based RNN-T, enhancing streaming speech recognition by capturing both local and global audio contexts, outperforming existing models with lower computational complexity.

Contribution

The paper proposes ConvRNN-T, a novel model that augments RNN-T with convolutional encoders, improving accuracy and efficiency for streaming ASR.

Findings

ConvRNN-T outperforms RNN-T, Conformer, and ContextNet on Librispeech.

ConvRNN-T achieves higher accuracy with less computational complexity.

ConvRNN-T is suitable for on-device streaming speech recognition.

Abstract

The recurrent neural network transducer (RNN-T) is a prominent streaming end-to-end (E2E) ASR technology. In RNN-T, the acoustic encoder commonly consists of stacks of LSTMs. Very recently, as an alternative to LSTM layers, the Conformer architecture was introduced where the encoder of RNN-T is replaced with a modified Transformer encoder composed of convolutional layers at the frontend and between attention layers. In this paper, we introduce a new streaming ASR model, Convolutional Augmented Recurrent Neural Network Transducers (ConvRNN-T) in which we augment the LSTM-based RNN-T with a novel convolutional frontend consisting of local and global context CNN encoders. ConvRNN-T takes advantage of causal 1-D convolutional layers, squeeze-and-excitation, dilation, and residual blocks to provide both global and local audio context representation to LSTM layers. We show ConvRNN-T outperforms RNN-T, Conformer, and ContextNet on Librispeech and in-house data. In addition, ConvRNN-T offers less computational complexity compared to Conformer. ConvRNN-T's superior accuracy along with its low footprint make it a promising candidate for on-device streaming ASR technologies.