Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillation

TL;DR

This paper introduces ICKD, a novel knowledge distillation method that preserves inter-channel correlations to enhance feature diversity and homology, leading to state-of-the-art performance in vision tasks.

Contribution

The paper proposes a new inter-channel correlation approach for knowledge distillation, improving feature diversity preservation and achieving superior results in vision tasks.

Findings

Outperforms existing methods on ImageNet classification and Pascal VOC segmentation.

ResNet18 achieves over 72% Top-1 accuracy on ImageNet with ICKD.

Introduces grid-level inter-channel correlation for dense prediction tasks.

Abstract

Knowledge Distillation has shown very promising abil-ity in transferring learned representation from the largermodel (teacher) to the smaller one (student).Despitemany efforts, prior methods ignore the important role ofretaining inter-channel correlation of features, leading tothe lack of capturing intrinsic distribution of the featurespace and sufficient diversity properties of features in theteacher network.To solve the issue, we propose thenovel Inter-Channel Correlation for Knowledge Distillation(ICKD), with which the diversity and homology of the fea-ture space of the student network can align with that ofthe teacher network. The correlation between these twochannels is interpreted as diversity if they are irrelevantto each other, otherwise homology. Then the student isrequired to mimic the correlation within its own embed-ding space. In addition, we introduce the grid-level inter-channel correlation, making it capable of dense predictiontasks. Extensive experiments on two vision tasks, includ-ing ImageNet classification and Pascal VOC segmentation,demonstrate the superiority of our ICKD, which consis-tently outperforms many existing methods, advancing thestate-of-the-art in the fields of Knowledge Distillation. Toour knowledge, we are the first method based on knowl-edge distillation boosts ResNet18 beyond 72% Top-1 ac-curacy on ImageNet classification. Code is available at:https://github.com/ADLab-AutoDrive/ICKD.