Multi-head Knowledge Distillation for Model Compression

TL;DR

This paper introduces Multi-head Knowledge Distillation (MHKD), a simple method for neural network compression that uses auxiliary classifiers at intermediate layers to better align student and teacher models, improving performance.

Contribution

The paper proposes MHKD, a novel approach employing auxiliary classifiers for feature matching in knowledge distillation, enhancing model compression effectiveness.

Findings

MHKD outperforms previous methods on multiple image classification datasets.

Auxiliary classifiers facilitate better feature alignment despite differing internal feature dimensions.

The method is simple to implement and improves distillation results.

Abstract

Several methods of knowledge distillation have been developed for neural network compression. While they all use the KL divergence loss to align the soft outputs of the student model more closely with that of the teacher, the various methods differ in how the intermediate features of the student are encouraged to match those of the teacher. In this paper, we propose a simple-to-implement method using auxiliary classifiers at intermediate layers for matching features, which we refer to as multi-head knowledge distillation (MHKD). We add loss terms for training the student that measure the dissimilarity between student and teacher outputs of the auxiliary classifiers. At the same time, the proposed method also provides a natural way to measure differences at the intermediate layers even though the dimensions of the internal teacher and student features may be different. Through several experiments in image classification on multiple datasets we show that the proposed method outperforms prior relevant approaches presented in the literature.