Channel Distillation: Channel-Wise Attention for Knowledge Distillation

TL;DR

This paper introduces a novel knowledge distillation method called Channel Distillation, which uses channel-wise attention and a decay strategy to improve student network performance, outperforming existing methods on ImageNet and CIFAR100.

Contribution

The paper proposes a new distillation approach combining channel-wise attention, guided knowledge transfer, and loss decay to enhance student network training.

Findings

Achieves 27.68% top-1 error on ImageNet with ResNet18.

Student outperforms teacher on CIFAR100.

Outperforms state-of-the-art distillation methods.

Abstract

Knowledge distillation is to transfer the knowledge from the data learned by the teacher network to the student network, so that the student has the advantage of less parameters and less calculations, and the accuracy is close to the teacher. In this paper, we propose a new distillation method, which contains two transfer distillation strategies and a loss decay strategy. The first transfer strategy is based on channel-wise attention, called Channel Distillation (CD). CD transfers the channel information from the teacher to the student. The second is Guided Knowledge Distillation (GKD). Unlike Knowledge Distillation (KD), which allows the student to mimic each sample's prediction distribution of the teacher, GKD only enables the student to mimic the correct output of the teacher. The last part is Early Decay Teacher (EDT). During the training process, we gradually decay the weight of the distillation loss. The purpose is to enable the student to gradually control the optimization rather than the teacher. Our proposed method is evaluated on ImageNet and CIFAR100. On ImageNet, we achieve 27.68% of top-1 error with ResNet18, which outperforms state-of-the-art methods. On CIFAR100, we achieve surprising result that the student outperforms the teacher. Code is available at https://github.com/zhouzaida/channel-distillation.