Knowledge Distillation for Small-footprint Highway Networks

TL;DR

This paper enhances small-footprint highway neural networks for speech recognition by applying knowledge distillation from larger models, significantly improving accuracy while maintaining a compact size suitable for embedded devices.

Contribution

It introduces the use of knowledge distillation to train compact highway deep neural networks, improving their accuracy and bridging the gap with larger models.

Findings

Achieved significant accuracy improvements with less than 0.8 million parameters.

Narrowed the performance gap between compact HDNNs and large DNNs.

Demonstrated effectiveness on the AMI meeting speech recognition corpus.

Abstract

Deep learning has significantly advanced state-of-the-art of speech recognition in the past few years. However, compared to conventional Gaussian mixture acoustic models, neural network models are usually much larger, and are therefore not very deployable in embedded devices. Previously, we investigated a compact highway deep neural network (HDNN) for acoustic modelling, which is a type of depth-gated feedforward neural network. We have shown that HDNN-based acoustic models can achieve comparable recognition accuracy with much smaller number of model parameters compared to plain deep neural network (DNN) acoustic models. In this paper, we push the boundary further by leveraging on the knowledge distillation technique that is also known as {\it teacher-student} training, i.e., we train the compact HDNN model with the supervision of a high accuracy cumbersome model. Furthermore, we also investigate sequence training and adaptation in the context of teacher-student training. Our experiments were performed on the AMI meeting speech recognition corpus. With this technique, we significantly improved the recognition accuracy of the HDNN acoustic model with less than 0.8 million parameters, and narrowed the gap between this model and the plain DNN with 30 million parameters.