Knowledge Distillation from Non-streaming to Streaming ASR Encoder using Auxiliary Non-streaming Layer

TL;DR

This paper introduces a layer-to-layer knowledge distillation method for streaming ASR models, using auxiliary non-streaming layers and autoregressive predictive coding to improve performance over previous token probability distillation approaches.

Contribution

It proposes a novel layer-to-layer KD approach with auxiliary non-streaming layers and APC-based loss, enhancing streaming ASR accuracy beyond existing methods.

Findings

Significant WER reduction compared to previous distillation methods

Effective use of auxiliary non-streaming layers for feature alignment

Improved prediction of unseen future contexts in streaming ASR

Abstract

Streaming automatic speech recognition (ASR) models are restricted from accessing future context, which results in worse performance compared to the non-streaming models. To improve the performance of streaming ASR, knowledge distillation (KD) from the non-streaming to streaming model has been studied, mainly focusing on aligning the output token probabilities. In this paper, we propose a layer-to-layer KD from the teacher encoder to the student encoder. To ensure that features are extracted using the same context, we insert auxiliary non-streaming branches to the student and perform KD from the non-streaming teacher layer to the non-streaming auxiliary layer. We design a special KD loss that leverages the autoregressive predictive coding (APC) mechanism to encourage the streaming model to predict unseen future contexts. Experimental results show that the proposed method can significantly reduce the word error rate compared to previous token probability distillation methods.