Balance Divergence for Knowledge Distillation

TL;DR

This paper introduces Balance Divergence Distillation, a novel method that enhances knowledge transfer in neural networks by addressing the imbalance in negative and positive information during distillation, leading to improved performance.

Contribution

It proposes a new divergence-based approach using reverse Kullback-Leibler divergence to better model small probabilities in teacher networks, improving lightweight student network accuracy.

Findings

Achieves 1-3% accuracy improvement on CIFAR-100 and ImageNet.

Improves mIoU by 4.55% on Cityscapes dataset.

Effective across various computer vision tasks and distillation methods.

Abstract

Knowledge distillation has been widely adopted in computer vision task processing, since it can effectively enhance the performance of lightweight student networks by leveraging the knowledge transferred from cumbersome teacher networks. Most existing knowledge distillation methods utilize Kullback-Leibler divergence to mimic the logit output probabilities between the teacher network and the student network. Nonetheless, these methods may neglect the negative parts of the teacher's ''dark knowledge'' because the divergence calculations may ignore the effect of the minute probabilities from the teacher's logit output. This deficiency may lead to suboptimal performance in logit mimicry during the distillation process and result in an imbalance of information acquired by the student network. In this paper, we investigate the impact of this imbalance and propose a novel method, named Balance Divergence Distillation. By introducing a compensatory operation using reverse Kullback-Leibler divergence, our method can improve the modeling of the extremely small values in the negative from the teacher and preserve the learning capacity for the positive. Furthermore, we test the impact of different temperature coefficients adjustments, which may conducted to further balance for knowledge transferring. We evaluate the proposed method on several computer vision tasks, including image classification and semantic segmentation. The evaluation results show that our method achieves an accuracy improvement of 1%~3% for lightweight students on both CIFAR-100 and ImageNet dataset, and a 4.55% improvement in mIoU for PSP-ResNet18 on the Cityscapes dataset. The experiments show that our method is a simple yet highly effective solution that can be smoothly applied to different knowledge distillation methods.