Learning to Teach with Student Feedback

TL;DR

This paper introduces Interactive Knowledge Distillation (IKD), a novel framework where the teacher learns from student feedback to generate tailored soft targets, improving model compression in NLP tasks.

Contribution

The paper proposes a new interactive framework for knowledge distillation that enables the teacher to adapt based on student feedback, enhancing training effectiveness.

Findings

IKD outperforms traditional KD methods on NLP tasks.

The iterative training process improves student model accuracy.

Teacher adapts dynamically to student training progress.

Abstract

Knowledge distillation (KD) has gained much attention due to its effectiveness in compressing large-scale pre-trained models. In typical KD methods, the small student model is trained to match the soft targets generated by the big teacher model. However, the interaction between student and teacher is one-way. The teacher is usually fixed once trained, resulting in static soft targets to be distilled. This one-way interaction leads to the teacher's inability to perceive the characteristics of the student and its training progress. To address this issue, we propose Interactive Knowledge Distillation (IKD), which also allows the teacher to learn to teach from the feedback of the student. In particular, IKD trains the teacher model to generate specific soft target at each training step for a certain student. Joint optimization for both teacher and student is achieved by two iterative steps: a course step to optimize student with the soft target of teacher, and an exam step to optimize teacher with the feedback of student. IKD is a general framework that is orthogonal to most existing knowledge distillation methods. Experimental results show that IKD outperforms traditional KD methods on various NLP tasks.