BD-KD: Balancing the Divergences for Online Knowledge Distillation

TL;DR

BD-KD introduces a balanced divergence approach for online knowledge distillation that improves both accuracy and calibration of compact models without additional post-processing, suitable for edge devices.

Contribution

The paper proposes BD-KD, a novel online KD framework that balances divergence losses to enhance model calibration and accuracy simultaneously, eliminating the need for post-hoc calibration.

Findings

Improved calibration and accuracy across multiple datasets.

Effective sample-wise weighting of divergence losses.

Outperforms recent online KD methods.

Abstract

We address the challenge of producing trustworthy and accurate compact models for edge devices. While Knowledge Distillation (KD) has improved model compression in terms of achieving high accuracy performance, calibration of these compact models has been overlooked. We introduce BD-KD (Balanced Divergence Knowledge Distillation), a framework for logit-based online KD. BD-KD enhances both accuracy and model calibration simultaneously, eliminating the need for post-hoc recalibration techniques, which add computational overhead to the overall training pipeline and degrade performance. Our method encourages student-centered training by adjusting the conventional online distillation loss on both the student and teacher losses, employing sample-wise weighting of forward and reverse Kullback-Leibler divergence. This strategy balances student network confidence and boosts performance. Experiments across CIFAR10, CIFAR100, TinyImageNet, and ImageNet datasets, and various architectures demonstrate improved calibration and accuracy compared to recent online KD methods.