# The Role of Teacher Calibration in Knowledge Distillation

## TL;DR

This paper investigates how the calibration accuracy of teacher models influences the effectiveness of knowledge distillation, showing that reducing calibration error improves student performance across tasks.

## Contribution

It reveals the importance of teacher calibration in KD and proposes a simple calibration method that enhances KD performance across multiple tasks.

## Key findings

- Teacher calibration error strongly correlates with student accuracy.
- Calibrating the teacher improves KD performance across various tasks.
- The proposed calibration method is compatible with existing KD techniques.

## Abstract

Knowledge Distillation (KD) has emerged as an effective model compression technique in deep learning, enabling the transfer of knowledge from a large teacher model to a compact student model. While KD has demonstrated significant success, it is not yet fully understood which factors contribute to improving the student's performance. In this paper, we reveal a strong correlation between the teacher's calibration error and the student's accuracy. Therefore, we claim that the calibration of the teacher model is an important factor for effective KD. Furthermore, we demonstrate that the performance of KD can be improved by simply employing a calibration method that reduces the teacher's calibration error. Our algorithm is versatile, demonstrating effectiveness across various tasks from classification to detection. Moreover, it can be easily integrated with existing state-of-the-art methods, consistently achieving superior performance.