Understanding and Improving Knowledge Distillation

TL;DR

This paper provides a comprehensive analysis of knowledge distillation, categorizing the types of teacher knowledge and examining their effects on student training, leading to better understanding and diagnosis of KD's successes and failures.

Contribution

It introduces a hierarchical categorization of teacher knowledge in KD and systematically analyzes their impacts on student training dynamics.

Findings

Teacher's knowledge of the universe acts as regularization.

Domain knowledge influences class relationship encoding.

Instance-specific knowledge rescales gradients based on difficulty.

Abstract

Knowledge Distillation (KD) is a model-agnostic technique to improve model quality while having a fixed capacity budget. It is a commonly used technique for model compression, where a larger capacity teacher model with better quality is used to train a more compact student model with better inference efficiency. Through distillation, one hopes to benefit from student's compactness, without sacrificing too much on model quality. Despite the large success of knowledge distillation, better understanding of how it benefits student model's training dynamics remains under-explored. In this paper, we categorize teacher's knowledge into three hierarchical levels and study its effects on knowledge distillation: (1) knowledge of the `universe', where KD brings a regularization effect through label smoothing; (2) domain knowledge, where teacher injects class relationships prior to student's logit layer geometry; and (3) instance specific knowledge, where teacher rescales student model's per-instance gradients based on its measurement on the event difficulty. Using systematic analyses and extensive empirical studies on both synthetic and real-world datasets, we confirm that the aforementioned three factors play a major role in knowledge distillation. Furthermore, based on our findings, we diagnose some of the failure cases of applying KD from recent studies.