Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization

TL;DR

This paper introduces a trust regularization technique for knowledge distillation in RNN language models, significantly reducing model size while maintaining state-of-the-art performance.

Contribution

It proposes a novel trust regularization method to enhance knowledge distillation for RNNLMs, achieving smaller models without performance loss.

Findings

Model size reduced to one-third of previous best

Maintains state-of-the-art perplexity on Penn Treebank

Reduces RNNLM size to 18.5% in speech recognition without WER degradation

Abstract

Recurrent Neural Networks (RNNs) have dominated language modeling because of their superior performance over traditional N-gram based models. In many applications, a large Recurrent Neural Network language model (RNNLM) or an ensemble of several RNNLMs is used. These models have large memory footprints and require heavy computation. In this paper, we examine the effect of applying knowledge distillation in reducing the model size for RNNLMs. In addition, we propose a trust regularization method to improve the knowledge distillation training for RNNLMs. Using knowledge distillation with trust regularization, we reduce the parameter size to a third of that of the previously published best model while maintaining the state-of-the-art perplexity result on Penn Treebank data. In a speech recognition N-bestrescoring task, we reduce the RNNLM model size to 18.5% of the baseline system, with no degradation in word error rate(WER) performance on Wall Street Journal data set.