Improving RNN transducer with normalized jointer network

TL;DR

This paper introduces a normalized jointer network to reduce gradient variance in RNN transducer training, combined with enhanced encoder and predictor networks, leading to state-of-the-art speech recognition results on multiple datasets.

Contribution

The paper proposes a novel normalized jointer network to address gradient variance issues in RNN-T training and combines it with improved network architectures for better performance.

Findings

Achieved state-of-the-art CER on AISHELL-1 benchmark.

Reduced gradient variance in RNN-T training.

Improved recognition accuracy on large-scale industrial data.

Abstract

Recurrent neural transducer (RNN-T) is a promising end-to-end (E2E) model in automatic speech recognition (ASR). It has shown superior performance compared to traditional hybrid ASR systems. However, training RNN-T from scratch is still challenging. We observe a huge gradient variance during RNN-T training and suspect it hurts the performance. In this work, we analyze the cause of the huge gradient variance in RNN-T training and proposed a new \textit{normalized jointer network} to overcome it. We also propose to enhance the RNN-T network with a modified conformer encoder network and transformer-XL predictor networks to achieve the best performance. Experiments are conducted on the open 170-hour AISHELL-1 and industrial-level 30000-hour mandarin speech dataset. On the AISHELL-1 dataset, our RNN-T system gets state-of-the-art results on AISHELL-1's streaming and non-streaming benchmark with CER 6.15\% and 5.37\% respectively. We further compare our RNN-T system with our well trained commercial hybrid system on 30000-hour-industry audio data and get 9\% relative improvement without pre-training or external language model.