Deep LSTM for Large Vocabulary Continuous Speech Recognition

TL;DR

This paper presents a new training framework for deep LSTM models that enhances speech recognition accuracy and efficiency, enabling successful training of models with over 7 layers and effective knowledge distillation for online applications.

Contribution

It introduces a layer-wise training and exponential moving average framework for deep LSTM models, along with a transfer learning strategy using segmental MBR for improved training efficiency.

Findings

Deep LSTM models with over 7 layers outperform conventional training methods.

The proposed framework reduces character error rate by 14% with minimal accuracy loss during distillation.

Transfer learning with segmental MBR enables effective training on small datasets.

Abstract

Recurrent neural networks (RNNs), especially long short-term memory (LSTM) RNNs, are effective network for sequential task like speech recognition. Deeper LSTM models perform well on large vocabulary continuous speech recognition, because of their impressive learning ability. However, it is more difficult to train a deeper network. We introduce a training framework with layer-wise training and exponential moving average methods for deeper LSTM models. It is a competitive framework that LSTM models of more than 7 layers are successfully trained on Shenma voice search data in Mandarin and they outperform the deep LSTM models trained by conventional approach. Moreover, in order for online streaming speech recognition applications, the shallow model with low real time factor is distilled from the very deep model. The recognition accuracy have little loss in the distillation process. Therefore, the model trained with the proposed training framework reduces relative 14\% character error rate, compared to original model which has the similar real-time capability. Furthermore, the novel transfer learning strategy with segmental Minimum Bayes-Risk is also introduced in the framework. The strategy makes it possible that training with only a small part of dataset could outperform full dataset training from the beginning.