Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach

TL;DR

This paper introduces a hierarchical multi-task learning approach for cross-modal knowledge distillation from language models to acoustic models, improving speech recognition performance by leveraging different units and auxiliary outputs.

Contribution

It proposes a novel hierarchical distillation framework with auxiliary layers, enhancing knowledge transfer between language and acoustic models beyond existing label-interpolation methods.

Findings

Hierarchical distillation improves speech recognition accuracy.

Auxiliary output layers enhance knowledge transfer.

Method outperforms traditional label-interpolation distillation.

Abstract

The remarkable performance of the pre-trained language model (LM) using self-supervised learning has led to a major paradigm shift in the study of natural language processing. In line with these changes, leveraging the performance of speech recognition systems with massive deep learning-based LMs is a major topic of speech recognition research. Among the various methods of applying LMs to speech recognition systems, in this paper, we focus on a cross-modal knowledge distillation method that transfers knowledge between two types of deep neural networks with different modalities. We propose an acoustic model structure with multiple auxiliary output layers for cross-modal distillation and demonstrate that the proposed method effectively compensates for the shortcomings of the existing label-interpolation-based distillation method. In addition, we extend the proposed method to a hierarchical distillation method using LMs trained in different units (senones, monophones, and subwords) and reveal the effectiveness of the hierarchical distillation method through an ablation study.