Rethinking the Knowledge Distillation From the Perspective of Model Calibration

TL;DR

This paper analyzes how model calibration affects knowledge distillation, revealing that calibrating teacher models improves student performance, especially with larger teachers, by addressing over-confidence issues.

Contribution

It introduces a calibration perspective to understand knowledge distillation, demonstrating that simple calibration of teachers enhances student performance, particularly for larger models.

Findings

Calibrated teachers lead to better student models.

Over-confident teachers hinder effective knowledge transfer.

Calibration improves the correlation between teacher size and student performance.

Abstract

Recent years have witnessed dramatically improvements in the knowledge distillation, which can generate a compact student model for better efficiency while retaining the model effectiveness of the teacher model. Previous studies find that: more accurate teachers do not necessary make for better teachers due to the mismatch of abilities. In this paper, we aim to analysis the phenomenon from the perspective of model calibration. We found that the larger teacher model may be too over-confident, thus the student model cannot effectively imitate. While, after the simple model calibration of the teacher model, the size of the teacher model has a positive correlation with the performance of the student model.