Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models

TL;DR

This paper introduces the Oracle Teacher, a novel CTC-based teacher model that uses both source inputs and output labels to improve knowledge distillation, leading to better student models and faster training.

Contribution

The paper proposes a new Oracle Teacher model for CTC sequence models that leverages target information, with a training strategy to prevent trivial copying and enhance distillation effectiveness.

Findings

Improves student model performance in speech and text recognition tasks.

Reduces training time for teacher models significantly.

Demonstrates effectiveness of target-aware distillation in sequence models.

Abstract

Knowledge distillation (KD), best known as an effective method for model compression, aims at transferring the knowledge of a bigger network (teacher) to a much smaller network (student). Conventional KD methods usually employ the teacher model trained in a supervised manner, where output labels are treated only as targets. Extending this supervised scheme further, we introduce a new type of teacher model for connectionist temporal classification (CTC)-based sequence models, namely Oracle Teacher, that leverages both the source inputs and the output labels as the teacher model's input. Since the Oracle Teacher learns a more accurate CTC alignment by referring to the target information, it can provide the student with more optimal guidance. One potential risk for the proposed approach is a trivial solution that the model's output directly copies the target input. Based on a many-to-one mapping property of the CTC algorithm, we present a training strategy that can effectively prevent the trivial solution and thus enables utilizing both source and target inputs for model training. Extensive experiments are conducted on two sequence learning tasks: speech recognition and scene text recognition. From the experimental results, we empirically show that the proposed model improves the students across these tasks while achieving a considerable speed-up in the teacher model's training time.